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The  Existence  of  Smooth  Densities  for  the 
Prediction,  Filtering  and  Smoothing  Problems 

Brief  Outline  of  Results 


A  large  number  of  papers  have  been  produced  giving  results  of  both  practical  and 
theoretical  interest.  I  am  grateful  for  the  support  of  the  U.S.  Air  Force  office  of  Scientific 
Research  and  hope  it  is  pleased  with  what  I  have  done. 

In  1987,  I  realized  the  central  role  played  by  the  idea  of  a  stochastic  flow;  this  gave  rise  to 
a  series  of  papers.  The  more  theoretical  consider  martingale  representation  and  integration  by  parts 
in  function  space.  This  in  turn  gives  rise  to  more  elementary  proofs  of  some  results  in  the 
Malliavin  calculus.  These  techniques  are  reported  in  papers  [2],  [3]  and  [8]. 

I  then  realized  how  the  proofs  could  be  adapted  to  show  the  existence  of  densities  for 
filtering,  smoothing  and  prediction  problems,  [1].  In  collaboration  with  John  Baras  and  Michael 
Kohlman  I  used  the  techniques  of  stochastic  flows  to  obtain  results  in  stochastic  control  [4],  [9  ], 
[11].  In  particular  the  martingale  representation  result  was  applied  and  equations  for  the  adjoint 
process  obtained  [5]  Similar  results  were  obtained  for  jump  processes  and  reported  in  [7].  Work 
with  my  student  Allan  Tsoi  has  included  integration  by  pans  formulae  and  time  reversal  of  jump 
processes,  [16],  [20],  [25].  With  my  student  Hailiang  Yang,  results  have  been  obtained  on  the 
adjoint  process  in  panially  observed  control  problems  [21].  My  most  interesting  recent  work 
contains  new  equations  for  the  adjoint  process.  Using  martingale  representation  results  it  is  shown 
the  adjoint  process  satisfies  forward  and  backward  equations,  of  which  the  latter  is  most 
significant.  These  results  are  described  in  [17],  [19],  [24].  The  adjoint  process  appears  in  the 
minimum  principle  and  so  plays  a  central  role  in  determining  any  optimal  control.  Martingale 
representation  results  to  minimize  expected  risk  are  described  in  [14]  and  [26]. 

Full  details  of  results  obtained  during  the  contract  are  presented  in  the  papers. 
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Abstract.  Using  a  simple  martingale  representation  result,  a  partial  integration-by-parts  formula  is 
obtained.  Quoting  the  results  of  Bismut  and  Michel,  it  then  follows  that  under  Hdrmander’s 
conditions  on  the  coefficient  vector  fields,  the  filtering,  smoothing  and  prediction  problems  have  C* 
density  solutions.  The  paper  does  not  require  the  development  of  any  analysis  over  Wiener  space. 
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Key  words.  Filtering,  prediction,  smoothing.  Malliavin  calculus,  stochastic  differential  equations. 


1.  Introduction 

Following  Malliavin’s  remarkable  work  (8],  there  have  been  other  treatments  of 
the  Malliavin  calculus,  including  those  of  Bismut  [1],  Stroock  [11]  and  Norris 
[10].  A  particularly  readable  account  can  lie  found  in  the  paper  of  Zakai  [13].  In 
[2],  Bismut  and  Michel  developed  a  conditional  version  of  the  Malliavin  calculus 
to  show  the  existence  of  a  conditional  density  in  filtering  and  smoothing  prob¬ 
lems.  Other  important  applications  of  the  Malliavin  calculus  to  filtering  problems 
include  the  work  of  Cattiaux  [4],  Kusuoka  and  Stroock  [11]  and  Michel  [9]. 
Using  a  simple  and  natural  expression  for  the  integrand  in  a  stochastic  integral, 
the  authors  [5]  have  been  able  to  give  an  elementary  proof  of  the  existence  of  a 
density  for  a  diffusion  under  Hormander's  conditions  for  the  coefficient  vector 
fields.  The  homogeneous  chaos  expansion  of  the  random  variable  is  also  obtained 
in  [5].  The  objective  of  this  paper  is  to  present  a  conditional  version  of  the  results 
of  [5]  and,  following  the  exposition  of  Zakai.  simplify  some  of  the  results  of 
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London,  England. 
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Bismut  and  Michel.  In  particular,  a  conditional  integration-by-parts  formula  is 
obtained  without  using  any  functional  analysis  over  Wiener  space.  However,  the 
delicate  technical  sufficient  conditions  for  the  integrability  of  the  inverse  of  the 
Malliavin  matrix  are  not  discussed.  For  these  we  refer  to  Bismut  and  Michel  [2]. 

In  this  paper,  the  following  system  of  stochastic  differential  equations  is 
considered: 

dx  =  X0(jc,  y)  df  +  X,(jc.  y)  du>'  +  Xj(x,  y)  d B'  +  Xj(x,  y)h'{x,  y)  dt. 

dy  =  Y„(y)d/+  Y;{ y)  d B’  +  Y,ly)hl(x,y)dt. 

Here  tv  =  (w' . tv")  and  B  =  (B\ . . . ,  Bn)  arc  independent  Brownian 

motions.  The  process  x  represents  the  unobserved  signal  process,  while  y 
represents  the  observation  process.  If  {  Y,}  is  the  right  continuous,  complete 
filtration  generated  by  {y,}.  then  the  filtering  problem  discusses  E[x,|Y,].  the 
prediction  problem  discusses  E[x,  |  YJ  when  /,  and  the  smoothing  problem 
discusses  £[*,  |  Ys]  when  ss  i. 

Using  the  simple  martingale  representation  result  of  [5),  a  conditional  version 
of  the  Malliavin  calculus  is  developed  in  Section  4.  Suppose  T  s=  /  and  let  c  be 
any  smooth  function  on  Rd  with  bounded  derivatives  of  all  orders.  In  Section  5. 
we  show  that  if  the  inverse  of  the  conditional  Malliavin  matrix  M„  r  belongs  to 
LP(Q)  for  all  p.  1  5?/7<sc,  then 

\b\~M\ YrlUtffy)  sup  |c(x)| 

I  Id*  J  1  1CRJ 

for  all  multi-indices  a  =  («i - - ad),  where  K(y)  is  a  Yr-measurable  random 

variable  which  is  finite  a.s. 

This  inequality,  using  simple  Fourier  analysis,  implies  that  the  random  variable 
x,  has  almost  surely  a  conditional  density  given  Yr.  which  is  infinitely  differenti¬ 
able.  Using  Jensen's  inequality  we  can  immediately  deduce 

1  (*r,)  |  yJ|«  K'(y)  sup  |c(x)|. 

I  I"*  J I 

where  s?  I  or  /.  Therefore,  the  smoothing,  filtering  and  prediction  problems 
for  x,,  given  Y5,  have,  almost  surely,  smooth  conditional  density  solutions. 

2.  Stochastic  Flows 

We  recall  in  this  section  the  properties  of  stochastic  flows,  and  in  particular  those 
relating  to  Mower  triangular’  systems  obtained  by  Norris  [10].  Sec  also  Stroock 

[11].  Let  w,  =  (w! . w"),  r5*(),  be  an  n-dimcnsional  Brownian  motion  on 

(fi.  F,  P).  Write  {  F,)  for  the  right  continuous,  complete  filtration  generated  by  w. 

Suppose  X„.  Xi _ _  Xm  arc  smooth  vector  fields  on  [().»]  x  Rd,  all  of  whose 

derivatives  arc  bounded.  Then  from  Bismut  [I],  or  Carverhill  and  Elworthy  13], 
we  quote  the  following  result:  » 
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THEOREM  2.1.  There  is  a  map  f :  fl  x  [0, »)  x  [().  co)  x  Rd  —>  R‘‘  such  that  ( i )  for 
0  =3  s  sS  /  anti  x  e  Rd  £,,(.v)  is  the  essentially  unique  solution  of  the  stochastic 
differential  equation 


=  X.»(f.  £.,(x))dt  +  X,(/.  £.,(*))  da-;.  (2.1) 

with  =  x.  (Note  the  Einstein  summation  convention  is  used.) 

(ii)  For  each  to.  s,  t  the  map  £,.,(•)  is  on  P.d  with  a  first  derivative,  the 
Jacobian.  =  DSJ ,  which  satisfies 


dDz  t  -——(/,  &.,(x))DS',  dt  +  —f(t,  &.,(*))£>,.,  dwl 


(2.2) 


with  initial  condition  Ds_z  =  I.  the  d  x  d  identity  matrix. 

REMARKS  2.2.  This  result  is  proved,  as  quoted,  for  possibly  time  in¬ 
homogeneous  coefficient  vector  fields,  X,,  though  we  shall  not  use  this  general¬ 
ity.  Note  that  (2.2)  is  obtained  formally  by  differentiating  (2.1).  In  fact,  equations 
for  higher  derivatives  Hn^ldxn  are  obtained  by  further  differentiation.  However,  if 
we  consider  the  enlarged  system  given  by  (2.1)  and  (2.2),  the  coefficients  are  not 
bounded,  because  of  the  linear  appearance  of  D,.,  on  the  right  of  (2.2).  However, 
Norris  [10]  has  extended  the  results  of  Theorem  2.1  to  such  systems  with  time 
homogeneous  coefficients.  To  state  Norris’s  results,  we  first  define  a  class  of 
‘lower  triangular’  coefficients. 


DEFINITION  2.3.  For  positive  integers  a,  d.  dx,...,dk  write  S„(d . dk) 

for  the  set  of  X  e  C~(Rd,  Rd)  of  the  form 


1  x(,V)  \ 

lx'\ 

X(x)  = 

1  XV.  X2) 

for  x  ~ 

[jca 

{X(k)(x',x2,...,xk)i 

[xk 

where  Rd  is  identified  with  Rd<  x  •  ■  •  x  x’  e  Rd>  and  the  A'  satisfy 
!l^||s(o.N)=  sup(  Sup  j~rvl  v  sup 

i€KJ  (  I  +  |X|  )  KjKk  / 

<*»  for  all  positive  integers  N. 

Write  S(d . dk)  =  U0S0(d, . dk). 


(2.4) 


REMARKS  2.4.  Note  Equations  (2.1)  and  (2.2)  can  be  considered  as  a  single 
system  whose  coefficients  are  not  bounded,  but  are  in  S(d.  d2).  The  final 
supremum  on  the  right  of  (2.4)  implies  the  first  derivatives  of  An>  are  bounded. 

as  are  the  first  derivatives  Dj  in  the  ‘new’  variable  x'  of  Xli)(x' . x').  This 

means  X{>1  is  allowed  linear  growth  in  x'.  a  situation  illustrated  in  (2.2).  We  quote 
from  Norris  the  following  result. 
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I  HEOREM  2.5.  Let  X„.  A'i . X,„  e  Slt(d{ . dk).  Then  there  is  a  map 

dKSlx  [0. x  [0,  oo)  x  RJ  -*  RJ  such  that 

(/)  for  0  «  s  =£  /  and  x  6  R‘‘  <5  (to,  s.  t,  x)  is  the  essentially  unique  solution  of  the 
stochastic  differential  equation 

d.r,  =  X„(x,)  dt+  X,(x.)  d  w;  (2.5) 

with  x,  =  x. 

(»’)  for  each  u>,  s.  t  the  map  s,  t,  x)  is  C”  in  x  with  derivatives  of  all  orders 
satisfying  stochastic  differential  equations  obtained  from  (2.5)  by  formal 
differentiation. 

(iii)  sup  E[  sup  |  DNd>((o,  s,  u.  x)|''] 

^  C(p,  s,  t,  R,  N,  d\ . <4 ,  a,  ||X0||s(„i.v,, ....  ||  X„||<;<„.,v)).  (2.6) 

REMARKS  2.6.  Norris  proves  Theorem  2.5  by  induction  on  i.  Write  (2.5)  as  a 
system  of  stochastic  differential  equations  for  /=  1 , A; 


Ax',  -  Xftx! . xftdt  +  X'/’ix! . x')dw{. 

xi  =  x'eRd>.  (2.7) 


Suppose  the  result  is  true  for  /=! . /-I  and  write  X'/'fat.  s.  /,  x1)  = 

A'‘y,(xJ(w) . xr‘(w).  x').  Then  (2.7)  can  be  written  in  the  form 

d.t',  =  X„(s,  t.  x{)dt  +  X,(s,  i.  x{)dw; 

and  Theorem  2.1  applied.  \  difficult  step  is  establishing  the  result  for  /  =  1. 
However,  this  follows  by  a  stopping  time  argument,  which  is  essentially  the 
method  by  Bismut  [1],  Returning  to  the,  possibly  time  inhomogeneous,  situation 
of  Theorem  2.1,  consider  the  process  V  defined  by 

A  Vt.,  -  ~  V,.((“~r(L &Jx))~  £  {—-(I.  ^.,U)))‘d /- 

-  V,.,-T?U'&.l(x))dw'„  (2.8) 

with  V,s  =  /.  Then  by  applying  the  Ito  rule,  we  see  d(D,., V,.,)  =  0.  while 
DS.,V, .**/.  the  dxd  identity  matrix.  Therefore.  V,,  =  £>"{,  By  applying 
Theorem  2.5(iii)  to  the  system  given  by  Equations  (2.1),  (2.2)  and  (2.8).  we  have 

\D*j\  ~  sup  \Df.„\  and  |  V*_\  =  sup  | 

i^U< I 

are  in  U'(fl)  for  all  p<*.  Finally,  for  0=5  jsS  t.  recall,  by  the  uniqueness  of  the 
solution  of  (2.1): 


\-.v> 


£<i.rU>)  =  £.,(£n.,U<i))  =  &.,(x).  if  X  =  X..\, 
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Differentiating  (2.9) 

D„,  =  D  JX.s  (2.10) 

and 

Vlu=  (2.1 1) 


3.  Martingale  Representation 

Consider  a  stochastic  differential  system  with  coefficients  in  some  set  S.  as 
discussed  in  Theorem  2.5.  and  let  $u(x.  0)  be  its  stochastic  flow  solution.  For 
some  7*  >  ()  consider  a  real-valued  differential  function  c  for  which  the  random 
variable  c(£„  r(x„))  and  the  components  of  the  gradient  C{($i.r(Xo))  are  in¬ 
tegrate.  Let  M,  be  the  right  continuous  version  of  the  martingale 

E[c(S,tU>))|F,]. 

There  exist  several  proofs  of  martingale  representation  results;  see.  for  example. 
Bismut  [I]  and  the  references  given  there.  However,  the  following  proof  in  the 
Markov  case,  see  [5],  is  particularly  straightforward. 

THEOREM  3.1.  For  0=5  T,  MT  =  £[c(fo.r(*«))]+Jo  y,(s)dwj  where 

y,(s)  =  E[C{($,.tU>))£>o.t  I  F,]D,7.iX,(s,  £».,(*«>)). 

Proof.  It  is  well  known  that  M,  has  a  representation 

M,  =  M0+ f  y.Wdwj  (3.1) 

Jo 

for  some  predictable  integrands  y, .  Because  the  process  $>.r(*o)  is  Markov 

=  E[c(6.t(x))  I  F,]  (3.2) 

=  E,{c(f,.r(x))} 

=  V(r,  x),  say,  where  x  =  &.,(*<>)■ 

By  Theorem  2.5  and  the  chain  rule  c($.r(*))  is  differentiable,  in  fact  smooth,  in 
x.  The  differentiability  of  £,.,[c(£,tU))]  in  1  can  be  established  by  writing  the 
backward  equation  for  £-7-(x),  as  in  Kunita  [6].  Consequently,  applying  the  Ito 
role  to  V(/,  x ),  with  x  =  §i.t(*«)  we  have 

V(t.  £,.,(*„))  =  V(().  x„)  +  |  (~  +  Lvj  ds  + 
f'flV 

+  —  (s,  §».,(x,,))Xj(s,  fo.i(xo))dwi  (3.3) 

Jo  ox 


where 
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n  jl  I  d  /  m  .  1 

(I 

'  1  -  ,.,=  1  _  I  /tlx,  (IX, 


By  (he  uniqueness  of  the  decomposition  of  special  semimartingales,  comparing 
(3. 1 )  and  (3.3).  we  must  (rave  (as  is  well  known)  P 


and 


,1V 

~+LV  =  {) 
Us 


av 


yis)  ~  X7  (s>  &M)Xi(s,  §(.,(x„)). 

From  (3.2) 

,1V 

^=£[c<(6.t(x))D,.t|FJ 

so  bv  (2.10) 

y,(s)  =  £'[q($,.t(x„))  0„,T  |  rjDo.kX,(s.  fo.s(x«)). 

COROLLARY  3.2.  77*  «„/,  <„ 

COROLLARY  3.3.  A/ore.  in  particular, 
c(&i:r(x0))  —  £[c(for(x„))]  -h 

+ 1  £[cf(&.T(x<,))DIKr  I  Fs]  O.T'X.U  x„))  d  .  (3.4) 

rEy!^)l'4'i  F\,S  Senerated  hy  ,he  sel  of  stochastic  integrals  of  the  form 

Ilf  '  j)d®*;  '/,e  r.  ««  smooth  functions  of  s  and  w,  at  time  s 

with  hounded  derivatives  of  all  orders, 

Proof.  <r(w,}  is  generated  by  g(w,)  for  ge  CZ(R%  If  wc  apply  Theorem  3  I  to 
>he  proc«  «...  so  where  ,„c  J.JL  is  Ihc°dc„mv  , 

H[gw(w,)|  F,J=  £,,[f,v(»v,)J  =  ?{*,), 
where 

7(lVj)  =  (ri(K’>) . 7«( «%»  =  E,...[(iV(  HV) . *,„(  „,))). 

Therefore. 


X(  w,)  -  E[ g(  w,)]  +  f  y,(w,)dw'. 

j  n 
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where  the  -y;€  Ck(Rd).  Consequently,  a(w,)  is  generated  by  stochastic  integrals 
of  this  form.  Allowing  the  integrands  to  depend  on  s  we  that  F, .  which  is 
generated  by  iv, ,  for  is  generated  by  stochastic  integrals  of  the  form 

f»  y,{s.  w,)dH-i,  where  y,-e  CHIA00)*  Rn)- 


REMARKS  3,5.  So  far  we  have  considered  an  n-dimcnsional  Brownian  motion 
»■  =  (tv1,  . . . ,  wm)  and  a  state  vector  xeR1'.  Consider  now  a  larger  system: 

suppose  B  =  (B' . B ")  is  an  /t-dimensional  Brownian  motion,  defined  on  a 

probability  space  (fl.  F,  P),  which  is  independent  of  »v.  Write  {F,}  for  the  right 
continuous,  complete  filtration  generated  by  B,  and  {G,}  for  the  right  continuous, 
complete  filtration  of  flxfl  generated  by  F,  x  F, .  Consider  a  second  state  vector 
ye  R1’  and  a  stochastic  differential  system  defined  on  (Ct  xfl,  Fx  F,  P  x  p)  by 
the  equations 

dx,  =  Xo(x,.  y,)d/  +  XAx,.  y,)dw',+  Xfix,.  y,)dB', 

dy,  =  V0(y()d/+  Yj(y,)dBi,  (3.5) 

with  (x(0),  y(0))  =  (*<,,  y0)e  Rd  x  Rp.  We  shall  suppose  the  coefficient  vector 

fields  A'o, ....  Xm ,  Y() . Y„  are  such  that  the  coefficients  of  (3.5)  belong  to 

the  space  S,  so  that  Theorem  2.5  can  be  applied.  Note  that  in  (3.5)  the  process  y 
is  not  influenced  by  the  process  x. 

NOTATION  3.6.  Suppose  (x,  y)e  RJ  x  Rp  is  the  state  of  the  system  (3.5)  at  time 
s.  We  shall  denote  the  solution  flow  of  (3.5)  for  t  >  s  by  the  map 

{x,  y)-+U,.,U.  y),  yj.,(y». 


The  Jacobian  of  this  map  looks  like 


>%x,.,(x,  y),  y,.,(y)) 
3(jc,  y) 


fa(x,Jx,  y)) 
'  dx 


d(X;Jx,  y)j 

dy 

d(y3.i(y)) 

ay 


(3.6) 


Write  D,_,(x.  y)  for  the  ‘partial’  Jacobian  <l(xSJ(x,  y))ldx.  The  existence  of  the 
large  Jacobian,  and,  therefore,  of  its  components,  including  D,_,,  is  given  by 
Theorem  2.5. 

As  in  [2],  we  now  introduce  a  new  measure  on  (CtxCl,  Fx  F)  by  a  Girsanov 
change  of  density. 


NOTATION  3.7.  Suppose  h(x,  y)  =  (h'(x,  y), . . . ,  hn{x,  y))  is  a  smooth  function 
in  C*(RJ*pt  Rn)  with  bounded  derivatives  of  all  orders.  Define  the  real  valued 
process  L  on  fix Clx [0, x)1  x  Rd  x  Rp  by 


L,.,(x,  y) 


lAx,.u{x,  y),  y3.„(y))dB;u- 


|  h'(xs,u{x,  y).  y,.u(y))2dn|. 
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Then 

dLs.,(x.  y)  =  LSJ(x,  y)h'(xsAx.  y).  y,,,(x,  y))  dB',.  (3.7) 

with  L,A*<  y)=  I.  so  L  is  a  {G,}  martingale.  Furthermore.  L*,,  =  supUV(  Ltt.„  is  in 
every  space  I  =s  p  < ».  Because  li  is  bounded,  vve  also  have  that  (Li 7.1)*  = 

•s'iip„s/(L(i.L)  is  in  every  U'(Pl),  I  ^/?<*.  We  could  consider  the  flow  given  by 
the  combined  system  (3.5)  and  (3.7).  However,  for  the  moment  note  that  for 


LoAx,,,  y„)  =  L,„U„.  yt>)LxJx.  y), 
so  writing  L  =  L, Ks(x„,  y„)  we  have 


dLa., 

Tl 


=  jLs.,U,  y) 


(3.8) 


and 


au, 

;>x<) 


dL<,  i.t 

fix,, 


L,.,(x,  y)  +  Li,.s 


<1LS, 

<lx 


(3.9) 


with  a  similar  equation  for  <)Li,.,ldy„. 

DEFINITION  3.8.  Define  a  measure  Ph  on  (ftxH.  Fx  F)  such  that  its  restric¬ 
tion  to  G,  is  given  by 

dPh(o),  <I>)  =  UAxiu  y«)  dP(w)  x  dP(u>). 

Then  Girsanov's  theorem  states: 


THEOREM  3.9.  Under  Ph  die  process  B’  is  an  n- dimensional  Brownian  motion 
independent  of  w,  where 


Therefore,  under  the  measure  Ph ,  the  process  (xSJ ,  y,.,)  is  the  solution  of  the 
stochastic  differential  equation 

dx, .,  =  X„(x,.,,  ys.,)<lt  + Xi(x,'l,ysJdwi,  +  X,(x,.l,ys.l)(iB'/  + 

+  X,(xSil.  ys.i)h'(xSJ,ysJdi. 

dy, .i  =  Vo(y,.,)d/+  Yj(ys,,)dB'!  +  Yj(yXJ)h'(xs,,.  ys.,)dt.  (3.10) 

with  (x,.s,y,.,)  =  (x,y)e  RJxR''. 


REMARKS  3.10.  The  system  (3.10)  provides  a  natural  setting  in  which  to  discuss 
filtering,  smoothing  or  prediction  problems.  The  process  x,  represents  a  signal 
which  is  not  observed  directly,  instead,  one  observes  the  process  y,  which  is 
influenced  by  x,  through  the  process  h(x,,  y,).  Write  {  V,}  for  the  right  continuous, 
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complete  filtration  generated  by  y ,  and  E>,  for  expectation  under  I\ .  The  filtering 
problem  discusses  Eh[x,  |  Y,],  the  smoothing  problem  discusses  Eh[x,  |  Yr],  where 
/==  T,  and  the  prediction  problem  discusses  Eh[-r,  |  Yr],  where  T. 

In  this  paper,  using  the  techniques  of  the  Malliavin  calculus,  we  given  sufficient 
conditions  in  the  filtering,  smoothing  and  prediction  cases,  that  the  conditional 
distribution  of  x,  has  a  smooth  density. 


4.  Integration  by  Parts 

Suppose  ()<  T  and  let  Ui,.T(u>)  be  an  Fr  measurable  random  variable  of  the 
form  discussed  in  Lemma  3.4,  that  is 


r ' 

U,.r(w)=  |  y,(s.  Bs)dB',. 

Jo 


(4.1) 


where  ^eCt((0,“)x|?")  for  l=s/^n.  Consider  the  system  given  by  (3.5), 
(3.7)  and  (4. 1 )  on  (O  x  Cl,  FxF,  PxP): 

dx,.,  =  X0(x,_,,  y5.,)d i  +  X;{xlA,  yJ>t)du>;  +  X,(ss,,,  yJ.,)dB{. 

y«(y,.,)d/+  V}(y,.,)dB!.  (4.2) 

dLj.,  =  Ls.Ji'ix,',,  y,.()  dB{, 

dOL-^t.BJdB}. 

Then  Theorem  2.5,  with  (x,.s,  y,.,.  L,.s,  USJ  =  (x,  y,  1.0),  can  be  applied  to  (4.2) 
and  we  can  consider  the  associated  stochastic  flow.  Note  (7,.,  does  not  involve  x, 
y.  or  L,  and  if  (7(l.s  =  U  then 

Uu  =  L/  + 1  ?,(s.Bs)dB'.  (4.3) 

Also,  if  L  =  L«.j,  from  (3.8) 

L„.(  =  LLs_,(s.  y).  (4.4) 

I  HEOREM  4.1.  Suppose  0  <  t  ^  T  and  lei  c  be  a  C*  function  on  Rd  with 
bounded  derivatives  of  all  orders.  Then  for  any  square  integrable  predictable  process 
u(s)  =  (u,(s) . um(s)) 


E  £  Ua,rLo.rc(x(Kl(x„,  y„))  |  U;(s)  d  w[  j 

=  £  E^U„,rU). 7<t(xo.r(x,„  y<i))D(i.,  |  Do!,X,(s)u,(s) d s j  + 


m  r  m  .  r  T 

+  Z  £  W>.TU.7-L(7.rc(jr„.,U«„  y«»  •  D,,'.’ X((j)»,(s)  ds 

i* l  L  Jo 
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-  I  b\ UiKrUMxiKl{ JC«.  >-„))  f  Lo’  ^  D&XAsMs)  ds).  (4.5) 
..I  L  Jo  <)JC0  J 

Pr«o/.  First  recall  the  derivation  of  Theorem  3. 1  and  write  for 

V{s,  x.  y,  L.  U)  =  E[L/o.tLo.t(x«i.  yo)c(x0.,(x„,  y0))  |  GJ 

=  £[(U  +  Ut.r)LL,.r(x.  y)c(x,.,(x ,  y))  |  GJ  (4.6) 

=  Hj.x.y.i..uC( U  +  Us.t)LL,.t(x,  y)c(xs.,(x,  y))]. 

The  martingale  representation  result  is  obtained  by  writing  down  the  Ito  formula 
for  V,  and  the  derivatives  of  V  are  found  by  differentiating  the  conditional 
expectation  (4.5)  in  x,  y,  L  and  U.  Note  that  for  s>  t  the  derivative  of 
c(Xj,,(x,  y))  in  x  is  zero.  We,  therefore,  have 

Go,tG>.t(*o<  yo)c(x0.,(x„.  yo)) 

=  E[Uo.tU.t(xo,  yo)c(x0.,(x„,  y0))]  + 

+  [  E[U„.rUi-Cx(xxtJ(xtuyo))D(K,\ Gs]Do!,(X,dw,s  +  X,dB',)  + 

Jo 

+ e[  Uo.tUtcAxoAxo,  yo))  — ~r—  |  a]  y,  d  b>  + 

+  f  £[G0,t£j.tc(x„.,(x0,  yo)  |  C,]h' d  B{  + 

Jo 

+ 1  e[  UojL  (x.  y)c(x0.,(x0,  y0))|  Gs  j(X,  d  w',  +  X,  d£')  + 

+  £[  G„.t£  (x.  y)c(x(,,(x„.  y0))  |  G,]  V,  dBj  + 

+  [  £[U,.t£o.7<-(x0.,(x0,  y0))  |  G,]y,  dB'.  (4.7) 

J(l 

Taking  the  product  of  (4.7)  with  J(/  Wi(j)divJ,  because  tv  and  B  are  independent 
under  P  x  P,  we  htive 


£[  Go.  /  i-o.  ] 

rC(Xo.j)  «,(s)dtv(] 

Jo 

m 

Go.t£o.tci(xo.,)L>o.i  [  £><7.iX1(.v)«,(s)dj|  + 

i-l 

Jo  J 

+  V  f 

w  - 
1-1 

I  UotC(  X”  4  f  - — —  f  r,  j.tXffs^d'1 

.  ■  Jo  fix  '  '  '  '  J 

(4  8) 
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From  (3.9), 


^  D «.! - ^  L,.r(.r,  V ) D(7.‘ . 

<ljf  (Ia'o  <lx o 


Substituting  in  (4.8)  the  result  follows. 

NOTATION  4.2.  Write  *  for  the  transpose.  Furthermore,  write 


X.WfdH-i, 


and  recall  the  Malliavin  matrix,  [1],  [5],  (which  here  is  a  ‘partial*  Malliavin  matrix 
in  the  X,  vector  fields): 


Mo,7-  =  £  f  J?SiX,(s)X,(5)*(OSl)*-dj. 

»»l  Jo 

COROLLARY  4.3.  We  then  have  the  special  case  of  Theorem  4.1  obtained  by 
taking  u,(s)  -  (D(7.',X,(s))*: 


E[  U<>,rL<>.i<(xt>.,)  Ra.r] 

=  E[  Uu.tL,>'1C<{x,u)  D(KiM0.,}  + 

+  e\  Uo.tLu.tL,).tO(.xik,)  — — —  —  (4.9) 

L  <>xo  J 

—  E[  IVtL<>.tC(*o.<)A<>.tJ 

COROLLARY  4.4.  Equation  (4.9)  is  still  true  for  vector,  (or  matrix),  functions  c. 

REMARKS  4.3  The  gradient  cx  of  c  occurs  in  only  one  term,  so  (4.9)  is  an 
'integration  by  parts’  formula.  Suppose  g  is  a  second  smooth  function  with 
hounded  derivatives  of  all  orders.  Applying  (4.9)  to  the  product  c(x0.,)g’(x, >.,)  we 
have 


E  [  U()'tLo,tC(xO',)  g(xoj)  Ro.t] 

=  £[t4TL(i.r(oU oJgUu)  +  c(x(hl)gAx<,.,))  D<uM,u]  + 

[dL  1 

U„.rU,rEo.lrc(x„.l)8(x«.i)  ~~  Hi.r  j 

-  £[(7„.rL(».Tc(T().,)g(j:o,,)A<l.-,  ].  (4- 1  °) 

From  Lemma  3.4  the  random  variables  Uo.r  generate  Fr  so  (4.10)  can  be  written 
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B[  L,,.rC(xo.<)  g(x'o,( )  Rh.t  I  R'v  1 
=  E[L,>.r(f*(x,u)g(x(U)  +  cUo.dfMx,,.,)) DiUM„,,  |  F, ■)  + 

+  E^Lo.rL,T.rC(JC«.«)K(jfo.i)  H>.r  |  Fr  j 

—  E[Lo,TcU(u)g(xo.t)An.T  |  Ft]. 

Under  P  x  P,  Yr  <=.  Fr  so 

F[L<i.rc(xo.,)£(xU',) Rt)mT  j  Vr] 

=  E[La,r(cx(xo.,)s(Xo,,)  +  c{x„J)gx(tXo,l))Do.Mo.l  |  Vr]  + 

+  Fi)_TL(t}rc(xoj) f’(xoj)  Mq.t  |  Vrj 

~  E[hn<(xo,t)g(Xo,,)ho.T  |  Vt]. 

Now 

£fc[c(jrn.,)g(*«.«)Ko.r|  W J  =  EfLo.rC(x„.,)g(x«.()/?o.r  I  Vr](E[Lo.r  I  V,-])"1. 
Furthermore,  L<>.  r  >  0  a.s.;  therefore 
£[U.r|  y.r,<0°  a.s. 

Consequently,  dividing  by  £(/U>.r|  Vr]  we  have 

Eh[c(xIKl)x(x„.,)Ro.r  I  Vr] 

=  £fcr{cJ(^)./)g(JCo.,)  +  c(jc„.,)g,(xo.,))Do.,M0.,|  Yr]  + 

+  E\ L<7.TC(x,M)g(x,,,)  —  Mo.r  I  Vr] 
l  0x„  J 

—  Eh [c(X(>./)g(A(),i)A„,r  |  Vr).  (4.11) 

where  both  sides  are  finite  a.s. 

With  this  in  mind,  to  obtain  a  bound  for  the  conditional  expectation 
£h[c*(x,if,)  |  Vr]  we  would  like  to  take  g  =  M(Tj£>o.!  in  (4.1 1).  However.  D,,.,  and 
M( involve  the  past  of  the  process  £<>.,,  D„,,  and  M<>,(.  This  difficulty  can  be 
circumvented  by  considering  an  enlarged  system.  A  second  difficulty  is  that  the 
function  g(M,u ,  D(1,()  =  Mo,!  D, 7.1  does  not  have  bounded  derivatives.  However. 
D_1  =  V  is  given  by  (2.8).  Considering  g,(M.  V)  =  (M  + e)'1  V  for  e>0  and 
letting  £-*()  we  see  Equation  (4.1 1)  holds  for  such  a  g. 

NOTATION  4,6.  Let  d><m(o>.  w,  s.  f,  x.  y,  L,  U )  denote  the  flow  associated  with 
the  system  (4.2).  Write  for  the  Jacobian  associated  with  tliL  now.  Note  that 
among  the  components  of  D (JH  are  the  ‘partial’  Jacobian 
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-y)  _ 
fix 


and  the  gradient  dLSJ(x,  y)H>x.  Write 

=  8,.,  =  J  (D“LX,(u))*dwL. 

nt  +1  -W 

A ['!!  =  A,,  =  Z  L7i^  D7iXAu)X,(u)*(D;i)*  du, 
i=i  J.,  to, 

ni  /•  / 

M'J°>  =  M,,  =  I  D7.LX((h)X7(m)*(D;.L)*  dll. 

*  ■»  I  Js 

Then  the  system 

^•>*(*««  D^.R™.  M(m,A<0’) 


is  Markov  with  coefficients  in 

S(d  +  p  +  2,  c/  +  p  +  2  +  (i/  +  p  +  2)2. 

2d  +  p  +  2  +  (d  +  p  +  2)2, 2d  +  p  +  2  +  2(d  +  p  +  2)2,  1 ). 

The  results  of  Theorem  2.5  apply  to  this  system  and  its  (low  <pn).  Note  that  Mj.,  is 
the  predictable  quadratic  variation  of  the  tensor  product  of  ft*.,  with  itself.  Write 
X\l)  for  the  coefficient  vector  fields  of  the  w*  integrals  in  <£(1\  and  D\[]  for  the 
Jacobian  of  d><0.  Also  write 

f<(0<!i-,X,,"(u))*dw!< 

Jo 

and  Mi'J  for  the  predictable  quadratic  variation  of  the  tensor  product  of  R[\]  with 
R("}  which  we  shall  denote  by 

M<!»  R(J!!>. 

Tlien  define  <p(2,=  (</>a\  D(l),  R(l>,  Mu))  so  d><2)  is  a  Markov  process  for  which 
the  stochastic  flow  results  of  Theorem  2.5  hold.  Proceeding  in  this  way  we 
inductively  define  Xin>  for  the  coefficient  vector  fields  of  the  w*  integrals  in  <f)ln\ 

R("J-  (DiT' X\"\u))  dwL, 

Jo 

~  (Rln)®  R(m)s.i 
and 

=  (4>,n\  D{n\  R{n\  MM). 

Write  Vn  for  the  gradient  operator  in  the  components  of 
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THEOREM  4.7.  Suppose  c  is  a  bounded  C’  scular  function  on  RJ  with  bounded 
derivatives.  Let  g  be  a  possibly  vector ,  {or  matrix ).  valued  function  on  the  state 
space  of  </)*'"  such  that  0,  /))  a/irf  V  „g{<f>in\Q,  l))  are  both  in  some  L'’(Cl). 

Then 

Bh[c(xt,,)g{d>in)( 0,  /))®Rl!:V|  Yr] 

=  &[c,(*u)g(*“”<0,  t))D0j Mo,,  |  Yr]  + 

+  &[c(jr<u)Vig(*<">(0.  t))D\CjM\rJ\  Vr]  + 

+  Eh  [  Lofrd.xOj)g(d>M(0,  t))  ~ p  Mo.r  ]  Yr] 

-£‘hfc(^,,)«(<#>'")(0,  t))A,,r|  Vr]. 

Proof.  The  result  follows  by  applying  to  the  system  4>in)  the  techniques  used  to 
derive  (4. 1 1). 


REMARKS  4.8.  Theorem  2.5  implies 


sup|  D{$|,  sup|Mo"s’|, 


sup 

J«S/ 


dL()'S 

bx„ 


sup|A«iS| 

s<i 


are  in  L‘‘(fi  x  Cl,  P  x  P)  for  all  1  :Sp<oo  and,  therefore,  finite  a.s.  Wc  have 
already  noted  that 


sup|Do,i|  and  sup  Ldi 

5<l  J*l 

are  in  every  Lp( fl  x  ft,  P  x  P),  I  p  <  co.  To  write  out  the  above  results  in  terms 
of  D0.,,  dL o.,/(3x  and  higher  derivatives  involves  very  involved  calculations.  Even 
in  the  one  dimensional  case,  it  seems  better  to  introduce  the  sequence  of  flows 
(/><n).  Theorem  4.7  can  again  be  thought  of  as  giving  a  conditional  ‘integration  by 
parts’  formula  for  cx . 

COROLLARY  4.9.  If  M,7.r  is  in  some  f/fftx  ft,  Ph)  taking  g(<f>{'\0 , /))  = 
Mo.'Do.l  itt  Theorem  4.7  we  have 

|  Yr] 

~  Eh[c{x{t.,)M()!,Do'i®  R{,.t  I  Yr] 

-  £/i[c(a:„,)(V ,  g)(  Do., ,  Mo.,)Do.]Mo.)|  Yr] 

~  E;,[c(jr(i.,) Lo.VM,,,1, Do.!  “7 —  Mo.r  |  Yr]  + 

i)x„  J 


T  Eh[c(X()j)Mt)jD()',Aoj\  Yr]. 

Because  the  remaining  terms  are  integrable  and,  therefore,  finite  a.s.,  we  have 
proved  the  following  result: 
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THEOREM  4.10.  Suppose  I\  is  the  probability  measure  of  Definition  3.8  and 
(Xo.,,  yo.,)  is  the  solution  under  !\  of  (3.10).  Let  c  be  a  smooth  function  with 
bounded  derivatives  of  all  orders.  Then  if  M,7.r  is  in  some  Lp(fl  x  fl,  Ph) 

|Efr,Uu)|  V,-]|=s  K(y)  sup  |c(x)|.  (4.12) 

i  cK'1 

where  K(y)  is  YV- measurable  and  finite  a.s. 

REMARKS  4.11.  Condition  (4.12)  implies  that  the  random  variable  Xo.,(x,„  y») 
has  a  conditional  density  given  YV,  d(x),  x  e  Rd  for  almost  all  y.  (See  Maliiavin 
[8j  or  Zakai  [13].)  Now  for  any  s  =*  7\  Ys  c  Yr  .  So  by  Jensen’s  inequality,  from 
(4.12) 

|  £[c, (*,,.,)  |  y,]|^  K'(y )  sup  (c(x)j.  (4.13) 

XCff1 

Equation  (4. 1 3)  holds  for  s  =£  /  or  s  ==  t  so  the  prediction,  filtering  and  smooothing 
problems  for  the  random  variable  Xo.,(*o.  yo)  all  have  a  density  for  almost  all  y. 

The  remaining  question  concerns  the  existence  and  integrability  properties  of 
M(7.r ■  These  have  been  carefully  studied,  see  Bismut  [1],  Maliiavin  [8]  and 
Stroock  [11].  For  (x.  y)  e  Rd  x  Rp  write  TXiY  for  the  vector  subspace  of  Rd 
generated  by  the  vector  fields  Xi(x,  y), . . . ,  Xm(x,  y),  and  the  Lie  brackets  of 

Xi(x,  y) _ _  Xm(x,  y)  and  X|(x,  y), . . . ,  X„(x,  y),  where  each  bracket  contains 

at  least  one  of  the  vector  fields  Xj(x,  y), . . . ,  Xm(x,  y).  Then  in  Theorem  1.19  of 
[2]  Bismut  and  Michel  show  that  for  all  T>  0,  M,7.V  is  in  Lp(Ylxfl,  Ph)  for  all 
I  p  <  <x>  if  the  following  condition  H,  analogous  to  a  condition  of  Hormander  is 
satisfied: 

H :  Tw<1  is  equal  to  the  whole  of  Rd. 

As  Bismut  [1]  observes,  if  H  is  satisfied  at  (x((,  yo)  then  it  is  satisfied  in  some 
neighbourhood  of  (x0,  yo). 

Finally  recall  that  if  u  is  a  nonsingular  linear  map  of  Rd  to  itself,  then  the  map 
d>:u—>  h”1  has  a  derivative  <p'(u)  which  is  a  linear  map  on  the  space  of  linear 
maps  of  Rd  to  itself  given  by  •  h  -  -  u~'  •  h  •  u~\  Applying  this  to 
g(D0.(,  M0.,)  =  Mo.\Do\.  we  have 

Eh[cx(x0,)\  Yt ] 

=  Eh[c(x„.,)M^Dfi!,®Ro.T  I  Yr] 

-  Eh[c(xnJ M<7.',((V , M0.,)( D(l\ MS,'.!) M.7.1  D.7.!  |  YT) 

-  £Jc(xo.,)M,7.!D,7.:((VlDo.,)(D!,,.!M[),.!)D(7.!|  Yt] 

-  Eh  f  c(x0.,) LZ'rMZ', D7u  —  Mo.,  |  Yt 

L  <>x0 

+  £h[c(A-o,)Mo.!  Do.iAo.r  |  YV]- 


(4,14) 
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5.  Bounds  Jor  Higher  Derivatives 

To  show  the  conditional  density  of  x,,.,  is  differentiable,  in  the  prediction,  filtering 
and  smoothing  situations,  we  shall  obtain  bounds  for  higher  derivatives  of  the 
form: 

I  4S (*'•«>  I  Vvl|*K(y>  sup|r(jr)|.  (5.1) 

I  L<IX  Jl  J ,CK" 

where  0  <  t  T.  Here  a  =  (a . .  ad)  is  a  multi-index  of  nonnegative  integers 

and 

,y’  !)ni  (■)":  d"-* 

<lx"  <lx"<  <1xp  3x‘jJ 

Again,  if  0=S  T,  then  Jensen’s  inequality  applied  to  (5.1)  gives 

|e[~U«.,)|  vJk'(y)  sup|c(jr)|.  (5.2) 

I  Ubr  Jl  se  RJ 

A  well-known  argument  from  harmonic  analysis  (see  [10],  or  [12])  shows  that  if 
(5.2)  is  true  for  all  a  with  jot  j  =  a,  +  a2  +  ■  •  •  +  ad  n  where  it  2=  d  +  I ,  then  the 
random  variable  jc<u(*o.  yo)  has  a  conditional  density  d(x)  given  Ys,  which  is  in 
Cn~d~'{Rd).  That  is,  we  have  a  differentiable  conditional  density  in  the  predic¬ 
tion.  filtering  and  smoothing  situations. 

To  see  how  to  proceed  apply  Corollary  4.9  to  cx  in  place  of  c.  (If  preferred, 
Corollary  4.9  could  be  applied  to  just  one  partial  derivative  3cl3xk  in  place  of  c. 
However,  the  result  is  true  for  vector  functions  c.)  This  gives 

Eh(c„Uo,)|  Yr) 

=  Eh[cAxn.,)MZ\Do.\®  Ro.r  I  Yr) 

-  Eh[cAxo.,)(Vtg)(Dlu,  MoJDll.lMll)  |  Yr] 

~  Eh^cx(xo.i)L{)}rMo.\Duj  Mi.r  I  Yr  j  + 

+  £/,[c,(x(,()Mo.IDo.IA„.7'  |  Yr}.  (5.3) 

Consider  the  four  terms  on  the  right  of  (5.3).  Each  term  is  of  the  form 

Eh[cx(xtu)hM{'\ 0,  i).*'"(0,  T))|  Yt].  «  =  1,2.  3,4. 

For  each  such  h,  consider  a  function  hi  =  hjMZ'DTu  and  apply  Theorem  4.7  to  c 
and  hi  to  obtain 

Ehlcx{x o,)W«,,,(0.  0,  <£u>(0.  T))\  Yr] 

=  Eh[c(xo,)hii<!>l'\0,  /),  <£“>(0,  T))@  R|J.t  |  YT] 

“  Eh[cixo,)(Vn(i)hi)(4>{'\0, t),  0l,,(O,  T))D!,:)M{,:)|  Yr] 
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-  G.fcUu)^  T)hi)(  0<!'(O.  /).  </»•"(().  T))D\l'rMo'r  |  Y,] 

-  E„[Lo.!,  c(xo.I)/i1(</><"(<). /).  </>,u(0.  T))  M,,r  |  Yr] 

+  £,fc(x(,1)/i;(</>,,>(().  f).  *<'»«).  T))A„r  ]  Yr], 

Subslitufing  in  (5.3)  we  obtain  an  expression  on  the  right  which  involves  only  r 
and  not  its  derivatives.  This  procedure  can  be  repeated,  using  Theorem  4.7.  At 
each  stage,  to  replace  a  term  of  the  form 

Efc[Cl(*o.,)/i(<A,n,(QcO.  </>("'«) .  T))  |  Yr] 

by  one  involving  only  c  define  h  =  hMfi\  0(7.]  and  applying  Theorem  4.7.  Clearly, 
higher  powers  of  are  introduced  at  each  iteration.  However.  Hormander's 
condition  H  is  sufficient  to  ensure  that  M(7.!  is  in  every  Ph),  1  /;<*=, 

We  have,  therefore,  proved  the  following  result: 

THEOREM  5.1.  Suppose  condition  H  is  satisfied.  Then  the  random  variable 
x„,,{x„,  y„),  the  solution  of  the  signal  process,  has  a  conditional  density  given  Y,  for 
almost  all  y  which  is  in  C'rj(Rd)  for  s  &  t  and  s  ^  t.  That  is,  under  condition  H  the 
prediction,  filtering  and  smoothing  problems  have  a  smooth  density  solution. 
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Abstract:  Using  the  ito  differentiation  rule,  the  properties  of  stochastic  (lows  and  the  unique  decomposition  of  special 
semimartingales,  the  integrand  in  a  stochastic  integral  is  quickly  identified. 

Keywords:  martingale  representation.  Malliavin  calculus. 


Introduction 

It  is  well  known  that  a  martingale  with  respect  to  the  filtration  generated  by  a  rownian  motion  is  a 
stochastic  integral  of  the  Brownian  motion.  There  have  been  many  derivations  o.  ne  integrand  in  this 
representation;  see  Clarke  (1970/71),  Haussmann  (1979),  Bismut  (1981)  and  Davis  (1980).  In  this  note  we 
give  a  very  short  derivation  in  the  Markov  case  using  the  properties  of  stochastic  flows  and  the  unique 
decomposition  of  special  scmimartingales. 


Flows 

Suppose  w  =  (w1,.,.,  wn)  is  an  n-dimensional  Brownian  motion  on  a  probability  space  (12,  5? ,  P). 
Consider  the  stochastic  differential  equation 

dx(=/(r,  x,)di  +  o(t,  x,)  dw,  (1) 

for  i  >  0,  where  / :  [0,  oo)  X  R"  -*  Rn  and  a :  (0,  oo)  X  Rn  -*  Rn  X  Rn  are  measurable  functions  which  are 
three  times  differentiable  in  *  e  Rn.  Write  £,,(*)  for  the  solution  of  (1)  for  s  ^  f.  having  initial  condition 
£,  ,(*)  =  a:.  Then  from  the  results  of  Bismut  [lj  there  is  a  set  Nc  <2  such  that  for  w  S  /V  there  is  a  version 
of  £,_,(*)  which  is  twice  differentiable  in  .v  and  continuous  in  t  and  s. 

Write  z,  =  9£,  s/dx  for  the  Jacobian  of  the  map  Then  it  is  known  that  z,  is  the  solution  of  the 
linearized  equation 

d-,  =fAs<  xs)zs  ds  +  <?,(•?.  x,)zs  dw, 

with  initial  condition  z ,  =  I,  the  n  X  n  identity  matrix. 

Consider  T,  an  initial  condition  x0s R"  at  time  [  =  0  and  a  function  c(£o.r(-vo))  of  the  final 

position  of  the  trajectory.  Here  c  is  a  differentiable,  real  valued  function  on  Rn  such  that  c(£0.Ax o))  and 
c{(^0T(x0))  are  integrable.  Write  { F, }  for  the  right  continuous  complete  family  of  a-fields  generated  by 
o{ws:  s^t}. 
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Then  M,  =  £(c(£0  r(.v0))  |  F,)  is  an  {  F,  }-njartingale.  and  so  by.  for  example.  Theorem  12.33  of  [4],  M, 
has  a  representation 


M,  =  A/0  +  f\ dnv, 

A) 


where  y  is  an  {  F, } -predictable  process. 


Theorem 


y,  -  Es.x  c{(^o.r(A’o))  a(S'  £o.j(a'o)) 

=  £  c((SoA*o))d\l'X)  \F,  (%*)  '(s.SoJx „)). 

Proof.  For  0  41  4T  write  .x  =  ?0.r(Ao)-  so  tha>.  by  the  semigroup  property  of  the  stochastic  flows, 
£o.r(-vo)  =  ?<.r\£o.<(-v  o))  =  £i.r(-x)- 

Then 

M,  =  £[c(^.r(x0))  |  £,]  =  £[c(S,.r(.x))  |  F,\  =  £,.r[c(^r(.v))] 

=  K(r.  ,v).  say. 

As  noted  above.  £,r(.x)  is  twice  differentiable  in  x.  The  differentiability  of  £(c(£,  r(x))  |  £.)  in  t  can  be 
seen  by  writing  the  backward  equation  for  £,  r(.v)  as  in  Kunita  [7],  However,  ,x  =  £0,(.x0)  so  expanding 
F(r.  £0f(.t0))  b>'  the  1,0  rule 

v(<-  So./(Ao))  =  Mf  =  K(0.  'vo)  +  £(^-(^>  $o.#(*o)) +  ***'(*•  (Ao))j  d* 

+  ^(Xo))o(s.  $0.,(A'o))  dw,.  (3) 


n  0  n  02 

L-Y  f'-z — -i- i  Y  a''-z — r —  where  a (/,  x.)  is  the  matrix  00*. 

,  d.v,  *  ,  d*,d.x. 

However. 

3F(r.  ,v)  ,  n,  rl  ,,  ,  \\ 3Cf.T(-x)  ,  n 

— 91 —  =  £[9^(^.r(x))|£,J  =£  ct\£i.T\x)) — — \F, 

r*  /  /■  ,  XX9^.r('X)] 

“  £|.jr  C((to ,r(-vo))  g^  I- 

A f,  is  certainly  a  special  semimartingale,  so  the  decompositions  (2)  and  (3)  must  be  the  same. 
In  particular,  equating  the  martingale  terms  we  have 

y ,  =  £,.*  C(((o.r(xo))-^f~  °(s’  to.s(xo)) 

_  p\ r  (P  (  V  13  1  cl/  d£°.'  \  „/„  £  (  -  W 

~  gx  l  “,j\  9x  /  vuv-'o//- 
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Remarks 

As  there  is  no  bounded  variation  term  in  (2)  we  must  also  have  immediately: 

$0.,(xo))  +  Us(xo))  =  0  with  1^(0.  .v())  =  £(c(  £„./•(  .v0))j. 

The  techniques  can  be  extended  to  more  general  martingales  as  in  Davis  (1980).  Similar  techniques 
quickly  give  the  results  of  the  Malliavin  calculus  in  this  situation. 
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INTEGRATION  BY  PARTS,  HOMOGENEOUS  CHAOS 
EXPANSIONS  AND  SMOOTH  DENSITIES' 

Bv  Robert  J.  Elliott  and  Michael  Kohlmann 
University  of  Alberta  and  Universitat  Konstanz 

By  iterating  a  martingale  representation  result  a  homogeneous  chaos 
expansion  is  obtained.  Using  the  martingale  representation,  the  integration- 
by-parts  formula  of  the  Malliavin  calculus  is  derived  using  properties  of 
stochastic  flows.  The  infinite-dimensional  calculus  of  variations  is  not  re¬ 
quired. 


1.  Introduction.  Since  Malliavin’s  outstanding  breakthrough  [9]  there  have 
been  other  treatments  and  simplifications  of  the  Malliavin  caiculus,  including 
those  of  Bismut  [2],  Stroock  [11],  Bichteier  and  Fonken  [1]  and  Norris  [10].  In 
this  paper  we  apply  a  very  simple  representation  of  the  integrand  in  a  stochastic 
integral,  Theorem  3.1,  to  first  derive  the  homogeneous  chaos  expansion  of  a 
certain  random  variable.  An  integration-by-parts  formula  is  obtained  and,  if  the 
Malliavin  matrix  M  has  an  inverse  which  belongs  to  every  LP(R)  (a  condition 
guaranteed  by  Hormander’s  H,  hypothesis),  it  is  shown  the  diffusion  has  a 
smooth  density.  The  principle  simplification  in  this  paper  is  the  observation  that 
by  considering  an  enlarged  Markov  system  only  the  simple  stochastic  integral 
representation  of  Theorem  3.1  is  needed.  No  infinite-dimensional  calculus  is 
required. 

2.  Flows.  In  this  section  we  recall  some  definitions  and  properties  of  sto¬ 
chastic  flows  on  cf-dimensional  Euclidean  space.  Suppose  w,  =  (wf . «//"), 

0  <  t,  is  an  m-dimensional  Brownian  motion  on  (R,  F,  P).  Write  {F,}  for  the 

right-continuous  complete  filtration  generated  by  w.  Let  *0,  X . Xm  be 

smooth  vector  fields  on  [0,  oo)  X  Rd  all  of  whose  derivatives  are  bounded.  Then 
from  Bismut  [2]  or  Carverhill  and  Elworthy  [4]  we  quote  the  following  result. 

Theorem  2.1.  There  is  a  map  £:  R  x  [0,  co)  x  [0,  oo)  x  Rd  -*  R'1  such  that: 

(i)  For  0  <,  s  <  t  and  x  e  Rd  ,(x)  is  the  essentially  unique  solution  of  the 
stochastic  differential  equation 

(2-1)  di.'Ax)-X0(t,it't(x))dt  +  X,(t,illJx))dw:, 

with  $fit(x)  =  x.  (Note  the  Einstein  summation  convention  is  used.) 
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(ii)  For  each  w,  s,  t  the  map  ,(•)  is  Cx  on  R‘l  —  Rd  with  a  first  derivative, 
the  Jacobian,  d£h  t/dx  =  Ds  „  which  satisfies 

d  \  d  V 

(2.2) 


initial  condition  Dh  8  =  I,  the  d  X  d  identity  matrix. 

(iii)  If  (  =  d2^s  ydx2  is  the  second  derivative,  then 

d  X  dX 

dw*.'=  -jf{(^,.,(x))ws.ldt  +  -^{t,^l(x))w^ldw; 

(^) 

+  DS'tdt+-j^{t,^l(x))Df't®  D^'dw;, 

with  WS  S  =  Q<=  ( Rd  ®  Rd)  ®  Rd. 


Remarks  2.2.  Note  that  (2.2)  and  (2.3)  are  obtained  formally  by  differenti¬ 
ating  (2.1).  However,  if  we  consider  the  enlarged  stochastic  system  given  by 
(2.1)— (2.3)  for  (£„_  „  Ds  t,  Ws(),  the  coefficients  are  not  bounded.  Nevertheless, 
Norris  [10]  has  extended  the  results  of  Theorem  2.1.  to  such  systems.  To  state 
Norris’  results  we  first  define  a  class  of  “lower  triangular”  coefficients. 


Definition  2.3.  For  positive  integers  a,d,dv...,dk  write  Sa(d{,...,dk) 
for  the  set  of  X  e  Cx(Rd,  Rd)  of  the  form 


(2.4) 

*(*)- 

X(2\x\x2) 

,  for  x  = 

. 

. 

.X{l,){x',x2 . **) 

where  Rd  is  identified  with  Rd'  X  •  •  •  xRdt,  xJ  e  Rd/  and  the  X  satisfy 


(2.5)  HXllso./v,  =  sup 


\DnX(x)\ 


,7*  o47£n  0  +  W> 


sup  \DjX{J)(x)\  <  oo, 

1  sjsk  , 


for  all  positive  integers  N.  Write  S(cf1( . . . ,  dk)  =  U aSa(dv ...,  dk). 


Remarks  2.4.  Note  (2.1)— (2.3)  can  be  considered  as  a  single  system  whose 
coefficients  are  not  bounded  but  are  in  S(d,  d 2,  d s).  The  final  supremum  on  the 
right  of  (2.5)  implies  the  first  derivatives  of  X*"  are  bounded,  as  are  the  first 
derivatives  Dj  in  the  “new”  variable  xJ  of  Xu){xx,...,xJ).  This  means  XiJ)  is 
allowed  linear  growth  in  x1,  a  situation  illustrated  in  (2.2)  and  (2.3).  We  quote 
from  Norris  [10]  the  following  result. 


X'i  HW.  <|V.  V 
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Theorem  2.5.  Let  X0,  X„ ... ,  Xm  e  Sa{dv ...,  dk).  Then  there  is  a  map 
4>:  S2  X  [0,  oo)  X  [0,  oo)  x  R'1  —  R<{  such  that: 

(i)  For  0  <  s  <  t  and  x  £  Rd,  <6(w,  s,  t,  x)  is  the  essentially  unique  solution  of 
the  stochastic  differential  equation 

(2-6)  dr,  =  X0(x,)  dt  +  Xt(x,)  dwf, 

with  xh  =  x. 

(ii)  For  each  <o,  s,  t  the  map  $(u,s,  t,x)  is  C“  in  x  with  derivatives  of  all 
orders  satisfying  stochastic  differential  equations  obtained  from  (2.6)  by  formal 
differentiation. 


(iii)  sup  E 
(2.7) 


sup  \Dn$(u,s,u,x)f 

s£U£t 


< 


C(p,s,t,R,N,dlt. 


■  * »  ^ Of, ||Aoj|j,(ai  JY)»  ...»  ll'^mlls(o.  ,V))' 


Remarks  2.6.  Norris  proves  Theorem  2.5.  by  induction  on  j.  Write  (2.6)  as  a 
system  of  stochastic  differential  equations  for  j  -  1, . . . ,  k, 


(2.8) 


<*/  =  HJ)(x\, ...,xf)dt  +  X}'\x),. . . ,  x{)  dwf, 
x{  =  xj  e  Rd/. 


Suppose  the  result  is  true  for  1 . .  —  1  and  write  XfJ\u,  s,  t,  xJ)  = 

XfJ\x){ w), ....  x{~  '(u),  x1).  Then  (2.8)  can  be  written  in  the  form 


dxf  -  X0(s,  t,  x /)  dt  +  Xi(s,  t,  xf)  dwf 

and  Theorem  (2.1)  applied.  The  difficult  step  is  establishing  the  result  for  j  -  1. 
However,  this  follows  using  a  stopping  argument,  a  technique  employed  by 
Bismut  [2,  3].  Using  the  notation  of  Theorem  2.1,  the  following  result  is  well 
known. 


Lemma  2.7.  For  0  <  s  <  t  write  ,  for  the  solution  of 


dV. 


£*o 

Bt 


(L  £,.,(*)) 


with  Va  t  —  I.  Then  Dg  ,VS  ,  -  /,  the  d  X  d  identity  matrix. 

Proof.  Applying  Ito’s  rule  to  V6  ,DS  „  we  see  d(Vs  ,DC  f)  =  0.  However, 
V,.sDR't  =  I.  □ 
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Remarks  2.8.  An  application  of  Jensen’s,  Burkholder's  and  Gronwall’s  in¬ 
equalities  shows  that  sup, J,  sup,,. u s ,| Ws  „|  and  supssus(]VNU|  are  in 
Lp(£l)  for  all  p  <  oo.  Alternatively,  this  conclusion  follows  from  applying 
Theorem  2.5  to  the  system  (2.1)— (2.3)  and  (2.9).  For  0  £  s  <  t,  by  the  uniqueness 
of  the  solution  of  (2.1) 

£o,t{xo)  ~  £«./(£(>. s(*o)) 

—  £*, <(•*[)>  if  ^  —  £().«(■*•())• 

Differentiating  (2.10),  using  the  chain  rule, 

(2.H)  Au  =  A.,4>.» 

and 

(2.12)  W0.£  =  WS''(D0iS  ®  A).*)  +  D*.,W0'S. 

3.  Representation  and  series  expansion.  Suppose  0  <,  t  ^  T  and  £,u(.v0) 
is  the  solution  of  the  stochastic  differential  equation  (2.1).  Consider  a  real- valued 
twice  continuously  differentiable  function  c  for  which  the  random  variable 
c(£0.t(*o))  I*16  components  of  the  gradient  c{(£0  T(*0))  are  integrable.  Let 

M,  be  the  right-continuous  version  of  the  martingale 

E[c{Z0,t(xo))\F,}- 

We  then  have  the  following  representation  result. 

Theorem  3.1.  For  0  £  t  <;  T,  M,  -  E[c<£0  r(x0))]  +  /0'y,(s)  dw‘,  where 

Y,(s)  =  £[c{({0.r(x0))Z)0ir|FsJZ)0'iA:i(s,^olS(^c))- 

Proof.  It  is  well  known  (see  [5],  for  example)  that  any  F(-martingale  M,  has 
a  representation 

(3.1)  Mt  =  M0+  (‘y,(s)  dw‘, 

/o 

for  some  predictable  integrands  yr  Because  the  process  £0,(;t0) *s  Markov 
Mt  =  F[c(^0,T(xb))|Ff] 

(32)  -«[4r(*)W 

=  ^.,[c(«t.r(*))j 

=  V(t,x),  say,  where  x  =  £0  ,(xo)- 

By  the  chain  rule  and  Theorem  2.1,  c(£,  T(x))  is  differentiable,  in  fact  smooth,  in 
x.  The  differentiability  of  F[c(|,  T(x))|Ff)  in  t  can  be  established  by  writing  the 
backward  equation  for  S<i7-(x)  as  in  Kunita  [8].  Consequently,  applying  the  I  to 
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rule  to  V(t,  x),  with  x  =  £ 0  l(x0 ), 


V{t,  €„.,(*«))  »  m  *0)  +  +  Lv]  ds 


+  f  J~(s,  USxo))x,{s,  ^o,s(x0))  div^, 
•'()  OX 


where 


d  g  l  dim 

£ x‘—  +  -  £  £ W 

«"—  I  OX,  *  ,,jml\k-l 


cbc,  dXj 


By  the  uniqueness  of  the  decomposition  of  special  semimartingales,  comparing 
(3.1)  and  (3.3),  we  must  have  (as  is  well  known) 

dV 

+  LV  —  0, 


From  (3.2) 


so  by  (2.11) 


Y«(s)  =  -^r(s>£o.8(*o))^;(s.  £o, «(*<>))• 


E[c((tS'T(x))DS'T\Fs 


Y,(s)  =  ^ [^(i0,r(-ro))A).rlZ:’s]-Do':i-^l(s,  S0..,(^0))- 


Remarks  3.2.  Note  in  particular  the  representation 


c(£o.r(*o))  =  £[c(£o.r(*o))| 


+/>l  ct(£o,r(-*:o))A),r|Z''* }  D0.lXi{s,  lo.iC^o))  dwls. 


Theorem  3.1  can  be  extended  immediately  to  vector  (or  matrix)  functions  c. 
Finally,  it  seems  the  proof  of  Theorem  3.1  can  be  extended  to  the  non-Markov 
case  ([6]). 

3.3  Notation.  Write  £(0)  =  £  for  the  solution  flow  of  Theorem  2.1,  and 
Z>(0)  =  D  for  its  Jacobian  given  by  (2.2).  Write  £(l)  for  the  d  +  cf2-dimensional 
process  with  components  £(1)  =  (£<0>,  D(0>).  Write  D<1>  for  the  Jacobian  of  this 
d  +  c(2-dimensional  process.  Write  £(2)  for  the  process  £<2)  =  (£<’*,  D(l))  and  so 
on.  Then  £(n+l)  =  (£(/,),  D(n)).  Note  £(n)  is  a  process  for  which  the  stochastic  flow 
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results  of  Theorem  2.5  hold.  Write 


dc 


c(iWr(*o))  =  ^U«.rU«))A,.r, 

dc<" 


**(&(*?))  = 


and  so  on,  so 


coh 


3c(n) 


Note  the  initial  condition  at  0  for  the  variable  D(,1)  is  always  the  identity  matrix 
of  appropriate  dimension.  Write  X,(n)  for  the  vector  field  coefficient  of  w‘  in  the 
stochastic  differential  equation  defining  £<n>  and  abbreviate 

*,(n)(Mo?Kn)))  a sX}n>(s). 

Then  by  iterating  Theorem  3.1,  we  have  the  following  representation  of  the 
random  variable  c(|0  r(x0)). 


Theorem  3.4.  If  c  has  hounded  derivatives  of  all  order,  then  for  any  n, 
c(Ur(x0))  =  F4c(H0'T(x0))\  +  £  £[c<*>(^(4*>))] 


(3.5) 


a-i 


X  /or(  {'  •  •  ■  (foS"tyX"~'X*(Sn)  •  •  ■  )  AM(«)  du>: 


xD0"lX,(s)du/J. 


Proof.  From  (3.4) 
c(W(*o))  =  £[cUo.r(*o))]  + 

=  E[c]  +  £[c(l>] J TDq1sX,(s)  dwf 

+  jfT(  /£e [c«2>|F5J  /)<■'- ’AV'^s,)  c/<)d-1^(S)  dw{. 

The  result  follows  by  repeated  application  of  the  representation  of  Theorem 
3.1.0 
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Remarks  3.5.  In  principle,  it  is  possible  to  write  the  previous  expansion  in 
terms  of  D0  „  W0i,  and  higher  derivatives  of  the  diffeomorpliism  £0„  rather 
than  by  considering  higher  and  higher  dimensional  systems.  However,  this  gives 
rise  to  very  complicated  formulae.  Consider  the  case  when  d  —  1.  Then  £(l>  = 
(£!0>,  D{0))  is  two  dimensional  and  by  Theorem  3.1 


SkWlF,!  -£|c£({0>7.(x0))D0,r] 


(3.6) 


+  c£(£u.T(*))Wu,rZ)0.jFa  Xt(u)dw'u 


dx. 


+  JoE[c({tu,T(xo))Du>r\Fu]~-(u)  dw'u 


Here  we  are  writing  Z0,Axo)  ~  £u,r(*)>  where  x  =  £0,«(*o)>  and  A>,r  *  A/.rA 
where  D  =  D0  u.  Note  the  final  integral  in  (3.6)  is  a  result  of  differentiating  in 
the  D  variables.  Recalling  (2.11)  and  (2.12),  we  have 


(3.7) 


where 


(3.8) 


Here 


and 


£[c(,>(*(,,)|Fs]  -£[ct(«o.T(*o))J>o.t 

+  J0  ^[cu(^o.t(xo))^q,t 

+  <?«($<>. r(xo))^o, rl-^u] Do,lX,(u)  dw'u 

-  /os£[c^0iT(*0  ))A,.t!^]^:X«^(«)  d< 

d  X 

+  jyic^oMxo)H.T\K}D0:l~-(u)dw‘ 

=  5(ct({0lT(*0))A),7-l  +  /  Y/(W)2)  du>i, 

J  0 

yj(u, 2)  =  «(u,2,l)J*;i*7(«)  -  «(«,2,2)A:Xu*/(") 

dX, 

+  <x(u,2,2)Dq'u—(u). 


a(u,2,])  -  E  [c^(f0  t(x0))Dq t  +  0{(£o, /(*Q))^>,rl-^u 
a(u,2,2)  =  £  [c{(£o,T(-*o))A>,rl^u]- 


--  - 

■-  --  ; - 


(3.9) 
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Substituting  (3.7)  in  (3.4),  we  have 

cUo.r(-x'o))  ~  ® [c(£q, ((*(>))]  +  & [c<('£o.r(<xo))A>,r]  Jq  D0  lX,(s)  dw^ 

+  f^\J(u,2)dw^D0:lX,(s)div:. 

I.i  turn,  the  martingales  a(u,2, 1)  and  a(u, 2,2)  can  be  expressed  as  stochastic 
integrals.  Substituting  again,  we  have 

c(€o.t(x0))  =  E\c(^t(x0))]  +  E[ct(f„.T(jCo))A».T|j[7^:Jf.(s)rf“i 

■b  E  [ c£j ( £0- r (-xo )) K t  c(( i«. /( -xo ) )  ^o, t ] 

x  |T(  u )  dwj )  D~  ‘  X,  ( s )  dwl 

-  [  c€(€o.  7-(^o))Ak  7-  J  ^  )z>0“  *  Jf.( » )  rfw; 

(3-10)  +^[ccUo.r(*o))AKrlj^T(j^i^(«)^)A^!^f(«)rf«!: 

+  jf{ jf( /o°Y*(o,3)  d^jDoJX/u)  dwi 

+  Joy\(v,4)dlv^D^W0iUX/a)dw/ 

+  jf  (jTya(o,5)  . c^Ja^.U) 

Remarks  3.6.  Theorem  3.4  [or  (3.10)  in  the  one-dimensional  case]  indicates 
how  a  “Taylor  series"  expansion  for  the  random  variable  c(£0>r(x0l)  can  be 
obtained  as  the  sum  of  multiple  stochastic  integrals. 

The  coefficients  of  the  stochastic  integrals  are  functions  of  the  expected  values 
of  c(i0  r(x «))  an<^  derivatives,  and  the  Jacobian  D0  r  and  its  derivatives.  The 
integrands  in  the  multiple  stochastic  integrals  do  not  involve  c,  but  are  functions 
of  the  Jacobian  and  its  derivatives,  and  the  coefficient  functions  Xr  By  unique¬ 
ness  the  expansion  is -the  same  as  the  homogeneous  chaos  representation.  This 
expansion  can  be  used  to  investigate  variations  about  the  expected  trajectory 
and  large  deviation  problems  ([7]). 

Corollary  3.7.  Taking  c($0  T(*0))  =  ^0.r(xo)  e  R<l>  so  c(  =  Id>  the  d  x  d 
identity  matrix,  and  ci(  =  0,  (3.4)  gives 

£o,r(->:o)  =  ^[So.tC^o)]  +  fn  [ A),rl^sj-^o.*^i(s)  d-w*< 

with  corresponding  higher-order  expansions. 
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Lemma  3.8.  Write  *  to  denote  the  transpose.  Suppose  c  and  g  are  real-val¬ 
ued,  differentiable  functions  such  that  the  random  variables  c(|or(.r(l)), 
g(£o,r(xo))>  C{(£o,r(*o)>>  g((to.Axo))  are  in  L2(.Q).  Then 

E[c{to.Axo))g{UAxo))\ 

=  £[c(^0,r(^))]^[^o.r(^o))] 

m 

+  E  L]oe[c^o.Axo))Do.t\K] 

xD0:lXi(s)XlAs)D*;'E[gt(^AxA)\K\ ds  . 

Proof.  By  Theorem  3.1 

g(to.Axo))  =  ^|^(^o.t(^o))]  +  f*E[gt{to.Axo))Do,T\F»]D(;.lX,(s)dwf. 

The  result  follows  by  taking  the  expectation  of  the  product  with  (3.4).  (Note 

g*=g-)a 

Definition  3.9.  The  nonnegative  matrix 

m 

K.,=  £  f  D-'uX((u)Xrtu)D,%du 

i-  1 

will  be  called  the  Malliavin  matrix  for  the  system  (2.1).  Note  that  something 
similar  to  M0  T  appears  in  Lemma  3.8.  In  some  references,  [11]  and  [12],  the 
matrix  D0  xAf0 >tACt 's  called  the  Malliavin  matrix. 

4.  Integration  by  parts. 

Theorem  4.1.  Suppose  c  is  a  twice  continuously  differentiable  scalar  func¬ 
tion  such  that  c(%0T(x0))  and  c((£0 fT(x0))  are  square  integrable.  Then  for  any 
square-integrable  predictable  process  u(s)  =  (u,(s), . . . ,  um(s)), 

E  c(^_Axo))  dw‘ 

=  E^fct(io.X^o))A,T/TDo';«^.(s)wt(s)cfe  • 

i-i  l  Jo 

Proof.  Using  the  representation  (3.4), 

E  c((0  T(x0))fJ'ui(s)dwf 
m  r  T 

=  ZE[jQ  ^[ct(«o.r^o))^o.rl^]^o:^<(s)«1(s)cfe 
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and  by  Fubini’s  theorem  this  is 


=  Ze 


i-  i 


c<(£o.r(-vo))A>. r  ds 


□ 


Corollary  4.2.  The  result  is  still  true  for  vector -  (or  matrix-)  valued 
functions  c. 


Corollary  4.3.  Taking  each  u;(s)  to  be  (DfflX,(s))*,  we  have 
rT / 


E 


c(W*o))/  (A  ).lX{(s)y  dwf  =  ^(c£(|0.r(a:{)))A,,r^o.r  • 


Corollary  4.4.  Consider  a  product  function 

h(£o.r(x  o))  =  c(^o.r(A:o))^(^o.r(-co)) 
satisfying  the  conditions  of  the  theorem.  Then 

^[cUo,r(^)k(^.,(^))/r(00;^Y<(S))»^; 


(4.1) 


=  M(ct(W(*o))£(£o.,Uo)) 

+  c(^o.r('ro))^£(^o,r(-<:o)))^o,  r^o.r]  ■ 


Remarks  4.5.  What  we  would  like  to  do  in  (4.1)  is  take 

g  =  M0AA>.r> 

so  that  we  can  obtain  a  bound  for  c{.  However,  A>~r  ar>d  Mqt  involve  the  past 
of  the  processes  £0  T,  D0  r,  M0  T.  This  difficulty  can  be  circumvented  by  consid¬ 
ering  an  enlarged  system,  similar  to  the  technique  used  in  Section  3.  However, 
the  sequence  of  enlarged  systems  is  different  to  that  discussed  in  Section  3,  so 
different  notation  will  be  used.  Note  that  even  when  the  original  process  £  is  one 
dimensional  the  method  leads  to  a  discussion  of  higher-dimensional  processes,  so 
not  much  simplification  is  obtained  by  taking  d  =  1. 

4.6  Notation.  Write  <£<0i(«,  s,  t,  x)  =  £s  ,(*)  for  the  stochastic  flow  defined 
by  (2.1).  Now  Dj°>(x)  =  D$  t(x)  denotes  the  Jacobian  of  the  flow  <£<0>.  From  (2.11), 
D  =  Au  and  *  =  ^(Jq), 

Af,(*o)  =  A»A 

so  the  system  (<p<0),  D<0>)  is  Markov.  Write  Rf\(x)  =  f‘(D~xuXt(u))*  dw'u  and 
R  =  /?<0°>s.  Then  R$,  =  R  +  D~  lR^t(x),  so  the  system  (4><0),  Z>‘0),  /?(0>)  is  Markov. 
Finally,  recall  the  definition  (3.9)  of  Ms  ,  and  write  Ms<0)  =  „  M  = 

Then  M^)  -  M  +  D~lMs  ,(x)D*~]  and  the  system 
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is  Markov  with  coefficients  in 

S{d,d  +  d\2d+  d\2d  +  d2). 

Consequently,  Theorem  2.5  applies  to  this  system  and  its  stochastic  flow  <pa>. 
Note  that  Mt  ,  is  the  predictable  quadratic  variation  of  the  tensor  product  of 
R,'t  with  R*r  Write  Xfl)  for  the  coefficient  vector  fields  of  w‘  in  Further¬ 
more,  write  4(1>  for  the  Jacobian  of  <£(I),  R['\  =  ff(D^l~lXfl\u))*  dw‘u  and  Mf]\ 
for  the  predictable  quadratic  variation  of  the  tensor  product  of  Rff\  with  i?(s0)*, 
which  we  shall  denote  by 

Then  define 

0<2)  =  /)<*>,  R(»t 

so  <f>(2)  is  a  Markov  process  for  which  the  results  of  Theorem  2.5  hold.  Proceeding 
in  this  way,  we  inductively  define  4>(n+l)  =  Din),  R(n\  M{n)),  where  R,n)  = 
f,Wy~lXjn)(u))dwl  and  M<n)  =  (R{n)  ®  P<0>).  Write  Vn  for  the  gradient 
operator  in  the  components  of 

Theorem  4.7.  Suppose  c  is  a  bounded  CK  scalar  function  on  Rd  with 
bounded  derivatives.  Let  g  be  a  possibly  vector-  ( or  matrix-)  valued  function  on 
the  state  space  of  <f><n)  such  that g(<f>{n\0,  T,  x0))  and  V„g(<p(n>(0,  T,  ,r0))  are  both 
in  some  LP(Q).  Then 

£[c(<J,<°>(0,r))g(^n>(0,T))  ®  n0,r] 

-  £[(v0c)(^(0,  T))g{^{ 0,  T))Do  tM^t\ 

+  ^'[c(<#><0)(0, 7’))(vrt^)(^(n>(0,  T))D^"j-Mo"t]  ■ 

Proof.  Applying  Theorem  3.1  to  eg,  we  have 
c(</>(0,(0,  T))g(<ffn)(0,T)) 

=  E\c{<t><°\0,T))g(<ffn\0,T))} 

+  ^r£[(voc}(<#>(0)(0,  T))^(<J>(n)(0,  T))D0  r)Fs]  dwf. 

+  ^T^[c(^<0,(0,  T))(v„^)(<J>(n)(0,  T))Z)^|Fg]Z)(5")"1A,l<n)(s)  dwf. 

Taking  the  tensor  product  with  and  the  expected  value,  the  result  follows. 

□ 

Remarks  4.8.  To  write  out  the  preceding  result  in  terms  of  D0 1,  ,  and 

higher  derivatives  of  the  flow  involves  very  involved  calculations.  Even  in 
dimension  1  it  seems  better  to  introduce  the  sequence  <f>(n)  of  flows.  Note 
Theorem  2.5  implies  supa  £  sup,  s  ate  in  every  Lp(Sl).  Theorem  4.7 
is  an  integration  by  parts  formula  as  only  one  term  involves  the  gradient  of 
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derivatives  V0c  =  c(  of  c. 

Corollary  4.9.  Taking  g(4>{i)(0,  T))  =  M^xTD(~\-t  if  M(fxr  is  in  some  LP(Q), 
^  E  [c{(£o.t(jco))]  =  ^  [c(£<u(*o))M>.rA>,r  ®  ^l\r] 

-£[c(|0,TW)(v,g)(D0.T,M,jDi'»TM^]. 

Because  the  remaining  terms  are  integrable  we  have,  therefore,  proved  the 
following  result. 

Theorem  4.10.  Suppose  £ 0.,(x0 )  is  the  solution  of  (2.1)  and  c  is  a  bounded 
smooth  function  with  bounded  derivatives.  Then  if  Mq\  is  in  some  Lp((l), 

(4.3)  |£[c{(£o.t(*o))]  |  ^  K  sup  |c(x)|. 

xeK* 

Remarks  4.11.  It  is  well  known  that  inequality  (4.3)  implies  that  the 
random  variable  Z0  T(x0)  has  a  density  (see  Malliavin  [9]  or  Stroock  [11]).  The 
remaining  question  concerns  the  existence  and  integrability  properties  of  Mq  Xt. 
These  have  been  carefully  studied  (see  Malliavin  [9],  Stroock  [11]  and  Norris 
[10]).  In  fact,  it  is  known  that  M0"j-  is  in  Lp(&)  for  all  p  <  oo  if  the  following 

condition  H,  of  Hormander  on  the  coefficient  vector  fields  X0 . Xm  of  (2.1)  is 

satisfied. 


Condition  H,.  *„...,  Xm,[X„  Xj\,  for  i,  j  =  0,...,  m,  [X„[X,,  Xk]]  for 
i,j,k  =  0,...,m,  etc.  evaluated  at  x0  s  Rd  span  Rd. 

Finally,  recall  that,  if  u  is  a  nonsingular  linear  map  of  Rd  to  itself,  then  the 
map  <(>:  u  -*  u-1  has  a  derivative  <p'(u),  which  is  a  linear  map  on  the  space  of 
linear  maps  of  Rd  to  itself,  given  by  <t>'(u)h  =  -u~lhu~l.  Applying  this  to 
g(A),r.  Mo.t)  =  Mo.tdo.T’  we  have 

E  [c{(^o,t(xo))] 

==  E  [c($0t7-(Xo))M0,7-I?o,7'  ®  ^O.V] 

(4.4) 

+  ■£  [  c{  £o,  t  ( *o ) ) M0. r ( ( V XM0. T){ ) M0  \ -D0  lT ] 
■i-5[c($0  T(x0))Mo  jD0  r((  VXD0 .r)(^o!r^o!r))A>.r]  • 


5.  Bounds  for  higher  derivatives.  To  show  the  density  of  io.rC^o)  *s 
differentiable,  we  must  obtain  bounds  for  higher  derivatives  of  the  form 


(5.1) 


E  t4(*o.t(*o))1|  ^  K  SUP  lc(*)l- 

.u'>  Jl  xer 
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Here  a  =  (a,, . ad)  is  a  multiindex  of  nonnegative  integers  and 

ga  go  i  gat  go,, 

In  fact,  a  well-known  argument  from  harmonic  analysis  (see  [10])  implies  that  if 
(5.1)  is  true  for  all  a  with  |a|  =  a,  +  ■  •  •  +  ad  £  n,  where  n  >  d  +  1,  then  the 
random  variable  £0.t(xo)  has  a  density  d(x)  =  d(x„...,xd)  which  is  in 
Cn~d~\Rd). 

To  sse  how  to  proceed,  apply  (4.2)  to  in  place  of  c.  [If  preferred,  (4.2)  could 
be  applied  to  just  one  partial  derivative  dc/d%k  in  place  of  c;  however,  the  result 
of  Corollary  4.9  is  true  for  vector  functions  c.]  This  gives 

(5  2)  =  ^  [c«(£o.r(*o))M),rA).r  ®  ^o.rj 

"■^[cf(^o.r(A:o))('7i^)('®o.r>  ^o.t)Do,tMo.t\  ■ 
Consider  the  two  terms  on  the  right, 

(5.3)  E [c£($0,r(:l’o))J^o.rA).r  ®  ^o.rj 
and 

(5.4)  £|c,(^T(^))(V,g)(Z)o.r,M0.r)Z)<'V<^|. 

5.1  Notation.  Write  M  =  M0T,  D  =  D0T,  D(l>  =  D{0'\,  etc.  Let 
^1(4><I))  be  the  function  ®  RM~lD~'  and  g2{<^2))  be  the  function 

(V,g)(A  M)DwM0)M~'D~l. 

Applying  Theorem  4.7  to  c  and  g{,  we  have 

E  [c(^o.r(a:o))^i(<#>(1))  ®  fllo!V] 

(5.5)  =  E [c£($0,r(;co))'^o.T-^o.7' ®  ^o.r] 

+  £[c(|0,T(x0))(v^J(4><2)(0,T))Z)^<V]- 

Applying  Theorem  4.7  to  c  and  g2  we  have 
[ c( ^o, 7'(-x:o))^2( ®  ^o,r] 

(5.6)  =  7'(x0))(Vi^)(/?o.T-^o.r)-^o.1/^o.lr] 

+  £[c(^o.r(xo))(v352)(«<3)(0,T))D(‘fVM('^]- 

Substituting  in  (5.2),  we  obtain  an  expression  on  the  right  which  involves  only  c 
and  not  its  derivatives.  This  procedure  can  be  iterated,  using  Theorem  4.7.  At 
any  stage,  to  replace  a  term  of  the  form  E[c((Z0  T(xo))h(<p{n\0,  T))]  by  one 
involving  only  c  define  h(<f>(n)(0,T))  -  /i(<£(n)(0,  T))Mq  \-Dq  \-  and  apply  Theo¬ 
rem  4.7.  Clearly,  higher  powers  of  M0‘ ]■  are  introduced  at  each  iteration.  [From 
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Theorem  2.5  Dff]-  is  in  every  L'\Q.). J  Hormander’s  condition  H,  is  sufficient  to 
ensure  that  MffxT  is  in  every  !/’(&),  1  <  p  <  so.  We  have,  therefore,  proved  the 
following  result. 

Theorem  5.2.  Suppose  Hormander's  condition  H,  is  satisfied.  Then  the 
random  variable  £0. r(-vo)  ^ias  a  density  d(x)  which  is  in  C^iR1*). 
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THE  PARTIALLY  OBSERVED  STOCHASTIC  MINIMUM  PRINCIPLE* * * § 

JOHN  S.  BARASf,  ROBERT  J.  ELLIOTTt,  and  MICHAEL  KOHLMANN5 


Abstract.  Using  stochastic  flows  and  the  generalized  differentiation  formula  of  Bismut  and  Kunita,  the 
change  in  cost  due  to  a  strong  variation  of  an  optimal  control  is  explicitly  calculated.  Differentiating  this 
expression  gives  a  minimum  principle  in  both  the  partially  observed  and  stochastic  open  loop  situations. 
In  the  latter  case  the  equation  satisfied  by  the  adjoint  process  is  obtained  by  applying  a  martingale 
representation  result. 

Key  words,  stochastic  control,  minimum  principle,  adjoint  process,  stochastic  flow 
AMS(MOS)  subject  classification.  93E20 

1.  Introduction.  Various  proofs  have  been  given  of  the  minimum  principle  satisfied 
by  an  optimal  control  in  a  partially  observed  stochastic  control  problem.  See,  for 
example,  the  papers  by  Bensoussan  [1],  Elliott  [8],  Haussmann  [11],  and  the  recent 
paper  [14]  by  Haussmann  in  which  the  adjoint  process  is  identified.  The  simple  case 
of  a  partially  observed  Markov  chain  is  discussed  in  the  University  of  Maryland  lecture 
notes  [9]  of  Elliott. 

In  this  article  we  show  that  the  minimum  principle  for  a  partially  observed  diffusion 
can  be  obtained  by  differentiating  the  statement  that  a  control  m*  is  optimal.  The  results 
of  Bismut  [5],  [6]  and  Kunita  [16]  on  stochastic  flows  enable  us  to  compute  in  an  easy 
and  explicit  way  the  change  in  the  cost  due  to  a  “strong  variation”  of  an  optimal 
control.  The  only  technical  difficulty  is  the  justification  of  the  differentiation.  As  we 
wished  to  exhibit  the  simplification  obtained  by  using  the  ideas  of  stochastic  flows, 
the  result  is  not  proved  under  the  weakest  possible  hypotheses.  In  §  6,  stochastic  open 
loop  controls  are  considered  and  a  similar  minimum  principle  with  an  explicit  adjoint 
process  is  derived  in  §  7.  If  the  optimal  control  is  Markov,  the  equation  satisfied  by 
the  adjoint  process  is  obtained  in  §  8  using  the  martingale  representation  result  of  [10]. 
This  simplifies  the  proof  of  Haussmann  [12].  Finally  in  §  9  it  is  pointed  out  how 
Bensoussan’s  minimum  principle  [2]  follows  from  our  result  if  the  drift  coefficient  is 
differentiable  in  the  control  variable. 

2.  Dynamics.  Suppose  the  state  of  the  system  is  described  by  a  stochastic  differen¬ 
tial  equation 

=f(l,  €„  «)  dt  +  g(t,  £)  dw„ 

U-l) 

£eR",  &=*o,  OStST. 

The  control  parameter  u  will  take  values  in  a  compact  subset  U  of  some  Euclidean 
space  Rk.  We  shall  make  the  following  assumptions: 
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(A,)  x0  is  given;  if  ,v0  is  a  random  variable  and  P0  its  distribution,  the  situation 
when  f -|.v|‘,P0(dx)  <00  forsome  q>  n  + 1  can  be  treated,  as  in  [14],  by  including 
an  extra  integration  with  respect  to  P0. 

(A2)  /:[0,  T]x  Rd  x  U Rd  is  Borel  measurable,  continuous  in  u  for  each 

(/,  x),  continuously  differentiable  in  x  and  for  some  constant  Kt, 
0+M)"l|/(t,.v,  n)|  +  |/v(f,x,  u)\SK{. 

(A.;)  g:[0,  7]x  R‘‘  -+  ®  R"  is  a  matrix-valued  function,  Borel  measurable,  con¬ 

tinuously  differentiable  in  x,  and  for  some  constant  |g(t,  x)j  +  |gx(f,  x)|g 
K2. 

The  observation  process  is  given  by 

(2.2)  dy,  =  h(t,)dt  +  di>t,  y,  s  R'",  y0  =  0,  Og/ssT. 

In  the  above  equations  w  =  (it'!,  •  •  • ,  w")  and  v  =  (u1,  •  •  • ,  v‘l)  are  independent 
Brownian  motions.  We  also  assume  the  following: 

(A4)  h :  R'1  -»  R"‘  is  Borel  measurable,  continuously  differentiable  in  x,  and  for 
some  constant  K 3,  |/i(r,x)|  +  |/rx(/,x)|giC!. 

Remark  2.1.  These  hypotlieses  can  be  weakened.  For  example,  in  (A4),  h  can  be 
allowed  linear  growth  in  x.  Because  g  is  bounded,  a  delicate  argument  then  implies 
the  exponential  Z  of  (2.3)  is  in  some  Lp  space,  1  <p  <co.  (See,  for  example,  Theorem 
2.2  of  [13].)  However,  when  h  is  bounded,  Z  is  in  all  the  Lp  spaces  (see  Lemma  2.3). 
Also,  if  we  require  /  to  have  linear  growth  in  u,  then  the  set  of  control  values  U  can 
be  unbounded  as  in  [14].  Our  objective,  however,  is  not  the  greatest  generality  but  is 
to  demonstrate  the  simplicity  of  the  techniques  of  stochastic  flows. 

Let  P  denote  Wiener  measure  on  C([0,  T],  Rn )  and  /x  denote  Wiener  measure 
on  C([0,  T],  Rm).  Consider  the  space  0  =  C([0,  T],  R")  x  C([0,  T],  R'n)  with  coordi¬ 
nate  functions  (tv,, y,)  and  define  Wiener  measure  P  on  Cl  by 

P(dw,  dy)  =  P(dw)fi(dy). 

Definition  2.2.  Write  Y -{Y,}  for  the  right  continuous  complete  filtration  on 
C([0,  T ],  R"‘ )  generated  by  T?=  cr{ys:  s^i}.  The  set  of  admissible  control  functions 
U  will  be  the  F-predictable  functions  on  [0,  T]x  C([0,  TJ,  R'")  with  values  in  U. 

For  ue  U  and  xe  R'1  write  £",(*)  for  the  strong  solution  of  (2.1)  corresponding 
to  control  u,  and  with  £"j(x)  =  x.  Write 

(2.3)  £”,(*)=  exp  (j  '  l>(Cr(x)y  dyr--\'  h(C,r(x)f  dr') 

and  define  a  new  probability  measure  P"  on  H  by  dP"/dP  =  Z^T(x0).  Then  under  P", 
(£o.t(xo),yi)  is  a  solution  of  (2.1)  and  (2.2),  that  is,  £o,,(x0)  remains  a  strong  solution 
of  (2.1)  and  there  is  an  independent  Brownian  motion  v  such  that  y,  satisfies  (2.2).  A 
version  of  Z  defined  for  every  trajectory  y  of  the  observation  process  is  obtained  by 
integrating  by  parts  the  stochastic  integral  in  (2.3). 

Lemma  2.3.  I !nder  hypothesis  (A4)  for  t  =L  T, 

EL(Zo'',(x o))p]<co  for  all  u  e  U  and  all  p,  lSp<oo. 


Proof. 


STOCHASTIC  MINIMUM  PRINCIPLE 


1281 


Therefore,  for  any  p  there  is  a  constant  Cp  such  that 


E[(z;,((.xo)nscp  1  +  e( 


(ZlXx 0))2h(a,{Xo))2  dr 


r/r 


The  result  follows  by  Gronwall’s  inequality. 

Cost  2.4.  We  shall  suppose  the  cost  is  purely  terminal  and  given  by  some  bounded, 
continuously  differentiable  function 


c(£o,r(xo))> 

which  has  bounded  derivatives.  Then  the  expected  cost,  if  control  i/e  U  is  used,  is 

./(«)  =  E„[c(£,'.T(x0))]. 


In  terms  of  P,  under  which  y,  is  always  a  Brownian  motion,  this  is 
(2.4)  J{u)  =  £[Zot(Xo)c(£o.t(*o1)]. 

3.  Stochastic  flows.  For  u  e  U  write 


(3.1) 


(“Ax)  =  x  + 


/(>',  (“Ax),  ur)  dr+ 


g(>\  (“Ax))  dwr 


for  the  solution  of  (2.1)  over  the  time  interval  [s,  t]  with  initial  condition  £",,(x)  =  x. 
In  the  sequel  we  wish  to  discuss  the  behavior  of  (3.1)  for  each  trajectory  y  of  the 
observation  process.  We  have  already  noted  that  there  is  a  version  of  Z  defined  for 
every  y.  The  results  of  Bismut  [5]  and  Kunita  [16]  extend  easily  and  show  the  map 


(l,\R"*Rd 


is,  almost  surely,  for  each  yeC([0,  T],  R"‘)  a  difleomorphism.  Bismut  [5]  initially 
gives  proofs  when  the  coefficients  /and  g  are  bounded,  but  points  out  that  a  stopping 
time  argument  extends  the  results  to  when,  for  example,  the  coefficients  have  linear 
growth. 

Write  ||^"(x0)||i  =  sup0£v=;,  |fo.s(Y0)|.  Then,  as  in  Lemma  2.1  of  [13],  for  any  p, 
1  §/;<oo,  using  Gronwall’s  and  Jensen’s  inequalities, 


orwitfsc  i+i*r+ 


g(r,  £>'.r(*o))  dwr 


almost  surely,  for  some  constant  C. 

Therefore,  using  Burkholder’s  inequality  and  hypothesis  (Aj),  ||f“(xo)lir  is  in  Ln 
for  all  p,  1  gp<oo. 

Suppose  u* s  U  is  an  optimal  control;  then  J(ti*)SJ{u)  for  any  other  ue  U. 
Write  £*,(•)  for  (“",(')■  The  derivative  d(*,(x)/dx  is  the  matrix  solution  C,  of  the 

fV» ••  o  i 

VVJUUUUU  IVI  O  l , 


(3.2)  dC, -fAU  (l(x),  u*)C,  dt+l  gAO,  (Ux))C,  dw\  with  Cs  =  I. 

»»  J 

Here  /  is  the  nxn  identity  matrix  and  g(n  is  the  /th  column  of  g.  From  hypotheses 
(A2)  and  (A3),/x  and  gx  are  bounded.  When  we  write  ||C||,  =  sup0S5S,  |C,|,  an  applica¬ 
tion  of  Gronwall’s,  Jensen’s,  and  Burkholder’s  inequalities  again  implies  ||C||r  is  in 
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Lp  for  all  p,  IS/) <oo.  Consider  the  related  matrix-valued  stochastic  differential 
equation 


D,  =  I  — 


(3.3) 


DJAr,  &(*),  “?)'  dr- l 


Drgx\r,  Zt,(x)Y  dw‘ 


-i  \'  Dr(g^(r,e.rM)')2  dr. 
i=  1  Js 


Then  it  can  be  checked  that  D,C,  =  I  for  t  ^  s,  so  that  D,  is  the  inverse  of  the  Jacobian, 
that  is,  D,  =  (<?£*, (x)/<5. v)-1.  Again,  because  /<  and  gv  are  bounded  we  have  that  ||£)||, 
is  in  every  Lp,  1 S  p  <  oo. 

Fora  d-dimensional  semimartingale  z,  Bismut  [5]  shows  that  £*,(z,)  is  well-defined 
and  gives  the  semimartingale  representation  of  this  process.  In  fact  if  z,  = 
Zj  +  A, +£"=,  J*  H,dw\  is  a  d-dimensional  semimartingale,  Bismut’s  formula  states  that 


TO 


&(*,)  =  *,+  |  [MtlAZr),  «?)  +  1  gx\r,  &U),  u*)?&(zr)H, 

i  si  I  ox 


(3-4) 


i  «  a2|*-(zr),„  „,\  , 

5  £rzr{H"H,)r 

—  dAr+  £  f*  (g(V,&(z,))  +  ^(z,)ft)  div). 
*  js  \  oX  I 


Definition  3.1.  We  shall  consider  perturbations  of  the  optimal  control  i<*  of  the 
following  kind.  For  s  e  [0,  T),  h  >  0  such  that  0  §  s  <  s  +  h  §  7;  for  any  other  admissible 
control  iie  U  and  Ae  Y,  define  a  strong  variation  of  u*  by 


u(t,  w)  = 


u*(t,  tv) 

a  (I,  >f) 


if  (t,  iii)ii[s,j  +  li)x/l, 
if  (/,  w)e[5,  s  +  /i]x  A. 


Applying  (3.4)  as  in  Theorem  5.1  of  [7],  we  have  the  following  result. 

Theorem  3.2.  For  the  perturbation  u  of  the  optimal  control  u*  consider  the  process 


(3.5) 


A  =  x  +  J  \f(r,  £,(*,),  Ur)  ~f(r,  &(*>,  «?))  dr. 


ft- 


% 


s 


Then  the  process  £*,(z,)  is  indistinguishable  from  £",(.v). 

Proof  Note  that  the  equation  defining  z,  involves  only  an  integral  in  time;  there 
is  no  martingale  term,  so  to  apply  (3.4)  we  have  H,  =  0  for  all  /.  Therefore,  from  (3.4) 


£*(z,)  =  -v  +  J  f(r,  £*r(zr),  uf)  dr 

,  fd£fr(Zr)\(dt;Zr(Zr)\~' ,  rr  „*  ,  „  ,  r/  „*  ,  ,  , 

+  J  y — “ — — — — J  (f(r,  £;,r(zr),  ur)  —f(r,  £‘r(z,),  ur ))  dr 


g(r,  ft(Zr))  dwr. 


However,  the  solution  of  (3.1)  is  unique  so  v 
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4.  Augmented  flows.  Consider  the  augmented  flow  that  includes  as  an  extra  coor¬ 
dinate  the  stochastic  exponential  Z*,  with  a  “variable”  initial  condition  ze  R  for 
Z*s(-).  That  is,  consider  the  (d  +  l)-dimensional  system  given  by 

£*,(*)  =  *  +  f  f(r,  Ztr(x),  u*)  dr+  g(r,  £*r(x))  dw„ 

J  s  j  s 

z*,(x,  z)  =  z  +  J  Z*r(x,  z)h(Cr(x)Y  dyr. 

Therefore,  from  the  first  equation  in  the  proof  of  Lemma  2.3  we  have 
Z*(x,z)  =  zZ*(: c) 

=  z  exp  /.(&(*))'  dyr-l~  j  fc(&(*))2  dr) 

and  we  see  there  is  a  version  of  the  enlarged  system  defined  for  each  trajectory  y  by 
integrating  by  parts  the  stochastic  integral.  The  augmented  map  (x,  z)-» 
(£*/(*),  Z*,(.y,  z))  is  then  almost  surely  a  difleomorphism  of  R'1’' '.  Note  that 
dg%(x)/dz  =  0,  df/dz  =  0  and  dg/dz  =  0.  The  Jacobian  of  this  augmented  map  is, 
therefore,  represented  by  the  matrix 

,(x)/dx 

z)/dx  dZ * 

and  for  l^i^d  as  in  (3.2) 

dik  dx, 

(4.1) 

dZUx,  Z)> 

dx i 


dZ 


*.(  &x)/dx  0  \ 

"  \dZUx,z)/dx  dZ*Mz)/dz)’ 
•2) 

7^=1  \'(z*r(X,z)- 
dx,  jes]  Js  \ 

+*'(&<*)) : 


dyi. 


(Here  the  double  index  k  is  summed  from  1  to  n.) 

We  shall  be  interested  in  the  solution  of  this  dilferential  system  (4.1)  only  in  the 
situation  when  z  =  l,  so  we  shall  write  Z*,(x)  for  Z*,(x,  1).  The  following  result  is 
motivated  by  formally  differentiating  the  exponential  formula  for  Z*,(x). 

Lemma  4.1. 

dZfAx )  Jf,  „  9£Z(X)  ,  \ 

— — ■ ■+,) 

where  v=  (v\  ■  •  ■ ,  v")  is  die  Brownian  motion  in  the  observation  process. 

Proof.  From  (4.1)  we  see  dZ'*,{x)/dx  is  the  solution  of  the  stochastic  differential 
equation 

(4.2)  d^XL  ('  ('^(x)  h,{0Ax))  +  zl(x)hAZUx)) dJ^)  dyr. 

,,(*)  =  Z*,(*)(J  hx '  ~~  ‘  dvr) 

dyr  =  h(£?Ax))  dt  +  dv,. 


J  s  \ 


Write 


where 
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Because 


the  product  rule  gives 


Z*l(x)«l  +  J,Z*(ac)A'(^r(jc))rfKr 


A,,(-v)  =  |  Zl(x)hx  •  ^  dvr  +  J  (  j  hx  ■  ^  •  dv^  Zl(x)h'(£jx))  dyr 
+  f '  ZUx)hU*r(x))-hx-^  dr 

S  0.v 

=  j'  L,r(x)hWAx))  (b’r  +  j  '  ^t(.V)/l,  "  ^  •  d)-r. 

Therefore,  LSJ(x)  is  also  a  solution  of  (4.2),  so  by  uniqueness 


Remark  4.2.  As  noted  at  the  beginning  of  this  section  we  can  consider  the 
augmented  flow 

(.V,  z)  ->  (&(*),  Z*,(.V,  2))  for  -T  6  R'1,  2  6  /?, 

and  we  are  only  interested  in  the  situation  when  2  =  1,  so  we  write  Z*,(.v). 

Lemma  4.3.  Z*,(z,)  =  Z",(x)  where  z,  is  the  semimartingale  defined  in  (3.6). 
Proof.  Z“,(x)  is  the  process  uniquely  defined  by 

(4.3)  Z‘f(x)  =  1  +  1'  Zl(x)h'(aM)  dyr. 

Consider  an  augmented  (d  + 1  )-dimensional  version  of  (3.5)  defining  a  semimartingale 
f,  =  (2,,  1),  so  the  additional  component  is  always  identically  one.  Then  applying  (3.4) 
to  the  new  component  of  the  augmented  process,  we  have 

ZfA Zr)  =  1  +  1  ZUzr)h'(t*ASr))  dyr 
=  1  +  |  Z*Azr)h'(eUx))  dyr 

by  Theorem  3.2.  However,  (4.3)  has  a  unique  solution  so  Z*,(z,)  =  Z",(.v). 

Remark  4.4.  Note  that  for  t>s  +  h 

Z*,(z,)  =  Z*(zs+„). 

5.  The  minimum  principle.  Control  u  will  be  the  perturbation  of  the  optimal  control 
u*  as  in  Definition  3.1.  We  shall  write  x  =  ^*Ax0).  Then  the  minimum  cost  is 

J(ii*)  =  £[Zo.r(A'o)c(fo.r(A'o))1 

=  E{ZUx0)ZtT(x)c{fi,T(x))l 

The  cost  corresponding  to  the  perturbed  control  u  is 

Hu)  =  E[Z*Axo)Z"'t(x)c(  £“  -,(*))] 

=  e[Z|t(Ao)Ztr(^/,)c(ftr(zJ+/())] 
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by  Theorem  3.2  and  Lemma  4.3.  Now  Z*r(  • )  and  c(££r(-))are  almost  surely  differenti¬ 
able  with  continuous  derivatives  and  z(,  given  by  (3.5),  is  absolutely  continuous. 
Therefore, 

J(u)~J(u*)  =  E[Zt(x0)(Z*T(zs+/,)c(^T(zs+(l))  -  Z*T(x)c(£*T(x)))] 


p  S'bh 

■■  E  y  V(s,  Zr)(/(r,  £*(zr),  »?)  -f{r,  (X),  «*))  rfr 


where  by  Lemma  4.1 


r(s,  zr)  =  Z0*s(x0)ZtT(zr){c,(^T(zr)) 

+  c(^r(zr))(|T/»^(zr))^(zr)  ^)](“(z,))  '■ 

Note  that  this  expression  gives  an  explicit  formula  for  the  change  in  the  cost  resulting 
from  a  variation  in  the  optimal  control.  The  only  remaining  problem  is  to  justify 
differentiating  the  right-hand  side. 

From  Lemma  2.3,  Z  is  in  every  Lv  space,  1  gp  <oo,  and  from  the  remarks  at  the 
beginning  of  §  3,  CT  =  dgfj/dx  and  Dr  =  (d$*T/dx)~l  are  in  every  Lp  space,  1  gp  <oo. 
Consequently,  T  is  in  every  Lp  space,  1  Sp<oo. 

Therefore, 

•j+/i 

J(u) -  J(u*)  =  E[(r{s,  zr)  -  V(s,  x))(/(r,  £*r(zr),  ur) -/(r,  £*,{zr),  «*))]  dr 

s 

+  f  +"  £[(T(.v,  x)-V(r,x))(f{r,  £*r(zr),  u,)-f(r,  £*r(zr),  «*))]  dr 

J  S 

+  j"5  B[r(r,  x)(/(r,  &(*,),  «,) -/(r,  £*r(zr),  u*) 

~f(r,  ^r(x),  ur)+f(r,  £j$r(x),  «?))]  dr 

ftbh 

+  j  E[F(r,  x)(f(r,  $r(x0),  ur)  -/(r,  &,(xo),  «*))]  dr 
=  f{h)  +  I2{h)  +  I3(h)  +  h{lt),  say. 

Now, 

•  j+/i 

|/t(/i )|S  K,  E[|f(s,  zr)-f(s,  x)i(I  +  Hr(Xo)iU)]  dr 

•»  5 

sk.a  sup  E[|r(i,zr)-r(s,x)|(i+||rU)||s+„)], 

i^rSt+/i 


*  5+ h 

Iz(h) | g  K2  £(|r(s,  x)  -  T(r,  s)|(l  +  H£“(*b)IU)]  dr 
SiKz/i  sup  E[|F(s,  x)  -  T(r,  x)|(l  +  ||fM(x0)||s  ,„)], 

sursis-^h 
*  s+h 

/3(/»)|SK,  £[|r(r,x)|  «&(*,) -£,(x)||]  rfr 

J  s 

=  EC3/i  sup  E[|r(r,x)|||^,(x)-^.(x)||5„1]. 

s&r$s+h 
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The  differences  |r(s;  z,)-T(s,  x)|,  |r(z, jr)-r(r, jc)|  and  ||f".(.v)-^*(.v)||i+)l  are  all 
uniformly  bounded  in  some  Lr,  />s  1,  and 

lim|r(5,zr)-r(vY)|  =  0  a.s., 

r-*s 

lim  |F(s.  .v)  -  r(r,  x)|  =  0  a.s., 

r-*s 

lim  B£(*)-£*(*)|U  =  0. 

/i-*0 


Therefore, 

lim  || !'(.?,  zr)-r(.T,.Y)||,,  =  0, 

r—  5 

lim  ||r(5,.Y)-r(r,.Y)||p  =  0,  and 

r—  v 


lim  ||(||^“.(.Y)-^t(A-)||SH.)|lP  =  0  for  some  p. 

/i-»0 


Consequently,  lim/.-o  /i  1  h(h )  =  0,  for  k  =  1,2, 3. 

The  only  remaining  problem  concerns  the  differentiability  of 


U  (/<)  = 


t )  h 


E{r(r,  x )(/(r,  ^r(.Vo),  «r)  -/(r,  £.r(x0),  «?))]  dr. 


The  integrand  is  almost  surely  in  L'([0>  7'])  so  lim,,-,,  h~'l4(h)  exists  for  almost  every 
xe[0.  T\.  However,  the  set  of  times  {s}  where  the  limit  may  not  exist  might  depend 
on  the  control  u.  Consequently  we  must  restrict  the  perturbations  u  of  the  optimal 
control  u*  to  perturbations  from  a  countable  dense  set  of  controls.  In  fact: 

(1)  Because  the  trajectories  are,  almost  surely,  continuous,  Yt,  is  countably  gener¬ 
ated  by  sets  {/I,,,},  i  =  1, 2,  •  •  •  for  any  rational  number  p  s  [0,  T],  Consequently,  Y,  is 
countably  generated  by  the  sets  {-4,,,},  pSt. 

(2)  Let  G,  denote  the  set  of  measurable  functions  from  (Cl,  Y,)  to  U  c  Rk.  (If 
u  e  U  then  u(t,  w)  e  G,.)  Using  the  L’-norm,  as  in  [8],  there  is  a  countable  dense  subset 
/•/,,  ={«,-,,}  of  G,,,  for  rational  pe[0,  T j.  If  H,  =  Ul>%ll  H/t  then  H,  is  a  countable  dense 
subset  of  G,.  If  uw  e  Hp  then,  as  a  function  constant  in  time,  ii„,  can  be  considered  as 
an  admissible  control  over  the  time  interval  [/,  T]  for  tl/i. 

(3)  The  countable  family  of  perturbations  is  obtained  by  considering  sets  A,„  e  Yn 
functions  uJf,e  H,,  where  pg  t,  and  defining  as  in  (3.1)  the  following: 


u%(s,  u-)  = 


H*(x,  w) 
un,(s,  w) 


if  (s,  »’)£(/,  T]xA„„ 
if  (j,  w)e[i,  T]x4,„. 


Then  for  each  /,  j,  p 


(5.1) 


lim  Ii 

h~0 


E[V(r,  x )(f(r,  £0*,(x0),  «/*)  -f(r,  $,(*„),  ,<*))]  dr 


exists  and  equals 

£[r(.v,  x)(f(s,  f*Jx0),  uJP)-f(s,  f*,(x0),  U*))1AJ 
for  almost  all  se[0,  T],  Therefore,  considering  this  perturbation  we  have 
lim  lr'(J(u%)  -J(u*))  =  E[V(s,  x)(f(s,  il(xo),  »,„) -f(s,  &(*>), 


SO  for  almost  all  .ve[0,  T]. 
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Consequently  there  is  a  set  Sc  [0,  T]  of  zero  Lebesgue  measure  such  that,  if  s£  S,  the 
limit  in  (5.1)  exists  for  all  i,j,  p,  and  gives 

E[V(s,  x )(f(s,  ft(.vo),  (t(x0),  u*))IAJ^ 0. 

Using  the  monotone  class  theorem,  and  approximating  an  arbitrary  admissible  control 
ue  V,  we  can  deduce  that  if  s&  S,  then 


(5.2)  £[I'(s,  x)(f(s,  t*,(x 0),  u) -f(s ,  &(*,),  u*))IA] is 0  for  any  Ae  Y,. 

Write 


pM-e* 


ce(&r(x 

oX 


+  c(^.r(.Vo))(  f  7  MfU*o))  dv„ 

\Js 


dX 


r.vM 


where,  as  before,  a'  =  £o,,(ao)  and  E*  denotes  expectation  under  P*  =  P“\  Then  p,(x) 
is  the  co-state  variable  and  we  have  in  (5.2)  proved  the  following  “conditional” 
minimum  principle. 

Theorem  5.1.  If  u* e  U  is  an  optimal  control  there  is  a  set  S<=[0,  TJ  of  zero 
Lebesgue  measure  such  that  if  s&S 


E*[PAx)f{s,  x,  i/*)!  K]S  E*[pfx)f{s,  x,  u)  |  TJ  a.s. 

That  is,  the  optimal  control  u*  almost  surely  minimizes  the  conditional  Hamiltonian  and 
the  adjoint  variable  is  pAx). 


6.  Stochastic  open  loop  controls.  We  shall  again  suppose  the  state  of  the  system 
is  described  by  a  stochastic  differential  equation 


(6.1)  d(,  =/(/,  £,  u)  dt  +  g(i,  £,)  dw,,  £,e  R‘‘,  £„  =  .y„,  0  S/if 

where  x0,f  and  g  satisfy  the  same  assumptions  A, ,  /U,  and  /\3  as  in  §  2. 

Suppose  tv  =  (»v‘,  •  •  ■ ,  tv")  is  an  n-dimensional  Brownian  motion  on  a  probability 
space  (f l,  F,  P),  with  a  right  continuous  complete  filtration  {FJ,  0  S|£l  Rather  than 
controls  depending  on  some  observation  process  y  we  now  consider  controls  that 
depend  on  the  “noise  process”  »v.  These  are  sometimes  called  "stochastic  open  loop” 
controls  [4], 

Definition  6.1.  The  set  of  admissible  controls  V  will  be  the  F, -predictable 
functions  on  [0,  T]xf2  with  values  in  a  compact  subset  V  of  some  Euclidean  space  Rl. 

Remark  6.2.  For  each  ue  V  there  is,  therefore,  a  strong  solution  of  (6.1)  and  we 
shall  write  £",(.v)  for  the  solution  trajectory  given  by 


(6.2) 


ft,(x)  =  x  + 


f(r,  £,(*),  «,)  dr  + 


g(r,  FJ(.y))  dwr 


Again,  because  u  is  a  (predictable)  parameter  the  results  of  [2],  (5],  or  [16]  extend  to 
this  situation,  so  the  derivative  d^'f/dx(x)  =  C“,  exists  and  is  the  solution  of 


(6.3) 


C”,-/  + 


ft(r,&Ax),u,)Cuvdr+  I 

k  —  I 


gfir^lA^Cldwf 


Suppose  D’l,  is  the  matrix-valued  process  defined  by 

D'lr  =  I  - 


(6.4) 


'  Dl,(f((r,  fr(x),  «,)-  gf'ir,  FJ(.v))2)  dr 
I  D-‘rgf  >(r,F‘ Ax))  dwKr. 


k  *1 


-  T* ' t-'1  zr-  ^  -  ----  -  -  - 
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Using  the  Ito  rule  as  in  §3  we  see  that  r/(D'',C“,)  =  0  and  D",C",  =  /,  so 

Di=(cirl. 


As  before,  if 


mx0)l=  sup  |ft,(*o)l, 


l|C“||T=  sup  |QS|,  ||D“||T  =  sup  |DS.,|, 

o  Sssr  o 

then  applications  of  Gronwall’s,  Jensen’s,  and  Burkholder’s  inequalities  imply  that 

lir(-v«)|L  IIC'll,.,  and  ||  D”  ||  r 
are  in  Lp  for  all  p,  1  Sp < °o. 

Cost  6.3.  As  in  §  2,  we  shall  suppose  the  cost  is  purely  terminal  and  given  by  a 
bounded  C2  function 


c{Co.t(xo))- 


Furthermore,  we  shall  assume 

|c(.v)|  +  |cv(.v)|  +  |c,,(.v)|^  KjO+M") 

for  some  q<co. 

The  expected  cost  if  a  control  u  s  V  is  used,  therefore,  is 

J(u)  =  £[c(&t(.y„))]. 

Suppose  there  is  an  optimal  control  i/*e  V  so  that 
;(#*)SJ(m)  foralli/eV. 

Notation  6.4.  If  u*  is  an  optimal  control,  write  for  £"*,  C*  for  C"‘,  etc. 

Deri  nition  6.5.  Consider  perturbations  of  m*  of  the  following  kind.  ForxefO,  T], 
h>0  such  that  0  £s<s+h^T  and  A  e  Fs  define,  for  any  other  li  e  V,  a  strong  variation 
of  u*  by 

,  IV(f,  >v)  if  (t,  H>)fS[syS  +  ll]XA, 
tu(f,  w)  if  (/,  H’)s[s,  s  +  /i]x  A. 


The  following  result  is  established  exactly  as  Theorem  3.2. 
Theorem  6.6.  For  any  perturbation  u  of  u*  consider  the  process 


(6.5) 


zr  =  x  + 


(fir,  t?A2r),  «,)  ~f(r,  tfr(2r),  »?))  dr. 


Then  the  process  gf,(z,)  is  indistinguishable  from  £“,(.v). 
Note  if  t  >s+h,  tfM)  =  &(*,+*)  = 

7.  An  open  loop  minimum  principle.  Now 

J(u*)  =  E[c(tir(Xo))l 

=  E[c(Z*t(x))) 

where  x  =  ^>5(x0). 

Similarly, 

J(u)  =  E[c(aT(x o))] 

=  H[c(|“t(x))] 

=  E[c(Ztr(zs+h))l 
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Therefore, 

J(u*)  =  E[c(f*r(st+h))-c(f*T(x))]. 

Because  £*r(  • )  is  differentiable  this  is 

=  E  |S+h  cMiriZr))  ^  (Zr)  •  U))  \f(r,  $l(zr),  i ir) 

(7.1)  ' 

-/('•,  &(r,),  «*))  dr  . 

As  in  §  5,  this  gives  an  explicit  formula  for  the  change  in  the  cost  resulting  from 
a  “strong  variation”  in  the  optimal  stochastic  open  loop  control.  It  involves  a  time 
integration  over  [s,  s  +  h]  and,  again,  the  only  remaining  problem  is  to  justify  the 
differentiation  of  the  right-hand  side  of  (7.1). 

Write 


1'U  r,  zr)  =  c((it.A-r))  —  ( -r ) (  —  (r,) 


»<(-v)  =  E  ct(£*  T{x0)) “ (x)\F, 

OX 


1  '  =  F[r(w)|FJ, 

where,  as  above,  „v  =  £;£,(.y0). 

Then  arguments  similar  to  those  of  §  5 — but  in  fact  simpler  because  Z  is  not 
involved — enable  us  to  show  that  there  is  a  set  Sc[0,  T ]  of  zero  Lebesque  measure 
such  that  if  st  S, 

E[\'(s,  s,  x)(f(s,  g$JXo),  u)-f(s,  tfjxo),  u*))lA]m 0 
for  any  »€  V  and  Ae  F,. 

That  is,  in  terms  of  the  adjoint  variable  pjx)  we  have  the  follov/ing  minimum 
principle  for  stochastic  open  loop  controls. 

Theorem  7.1.  If  V  is  an  optimal  stochastic  open  loop  control  there  is  a  set 
S  <=  [0,  T]  of  zero  Lebesgue  measure  such  that  if  si  S 

Ps(x)f(s,  x,  u*)  g  pfx)f(s,  X ,  u)  a.s. 

for  all  ue  V.  That  is,  the  optimal  control  u*  almost  surely  minimizes  the  Hamiltonian 
with  adjoint  variable  p,(x). 

Remark  1.2.  Under  certain  conditions  the  minimum  cost  attainable  under  the 
stochastic  open  loop  controls  is  equal  to  the  minimum  cost  attainable  under  the  Markov 
feedback  controls  of  the  form  u(s,  go  ,(x„)).  See  for  example  [3],  [12].  If  u„,  is  a 
Markov  control,  with  a  corresponding,  possibly  weak,  solution  trajectory  f",  then  i  tM 
can  be  considered  as  a  stochastic  open  loop  control  uM(w)  by  putting 

«A/(M')=«A/Uf o.YUo,  «'))• 


This  means  the  control  in  effect  “follows”  its  original  trajectory  rather  than  any 
new  trajectory.  That  is,  the  control  is  similar  to  the  adjoint  strategies  considered  by 
Krylov  [15].  The  significance  of  this  is  that  when  we  consider  variations  in  the  state 
trajectory  £  and  derivatives  of  the  map  x~>f,_,(x),  the  control  does  not  react,  and  so 
we  do  not  introduce  derivatives  in  the  u  variable. 
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(7.3) 


If  the  optimal  control  u*  is  the  Markov,  then  the  process  £*  is  Markov  and 

ps(x)  =  E[r(s,s,x)\Fs] 

=  E[r(s,s,x)\x]. 


8.  The  adjoint  process.  Suppose  the  optimal  stochastic  open  loop  control  m*  is 
Markov.  The  Jacobian  d|*r/d.v  exists,  as  does  (d$%T/dx)~l  and  higher  derivatives. 
Theorem  8.1.  Suppose  the  optimal  control  u*  is  Markov.  Then 


pJx)=E[ct(£*r(Xo))Cn,}- 


Pr(€SAxo))f({r,  8Ax o),  uf)  dr 


+  [  />*('•,  Elr(x0))g(r,  t£r(.x o))  dw, 

Jo 

-  [  Px(r,  tlr(Xo))g(r,  £tAxo))8((r>  £*. Axo))  dr. 
Jo 


Proof.  Write /{(;■)  for  f((r,  £t(.v0),  uf)  and  g(r)  for  g(r,  ££,(*<>)),  etc.  By  unique¬ 
ness  of  the  solutions  to  (6.1) 


(8.1) 


£o,r(-vo)  =  itrAKxA) 


so,  differentiating, 

(8.2)  Co'T  =  CS'tC0'S 

where  C0.r=  C *.r,  etc.  (without  the  *). 

From  (7.2)  and  (7.3) 

pAx)  =  E[c((aT(Xo))C,T\F,l 

so  from  (8.2) 

(8.3)  Ps(x)Co.<  =  F[c^„*r(.v0))C0.r|  FJ, 

and  this  is  a  (P,{F,})  martingale.  Write  x  =  £*Axo),  C-C 0.,.  From  the  martingale 
representation  result  [10],  the  integrand  in  the  representation  of  ps(x)C  as  a  stochastic 
integral  is  obtained  by  the  ito  rule,  noting  that  only  the  stochastic  integral  terms  will 
appear.  These  involve  the  derivatives  in  x  and  C.  In  fact,  by  considering  the  system 
with  components  £*.<  and  C0>/  and  any  real  C2  function  <t>,  the  martingale 

Ms  =  E[4>(£0,tIFs]=  E[<l>(i<KT)lx,  C]=  V(s,x,  C) 

=  V(0,  ,V0,  I)+  I  Vx(r,  €lr(Xo),  Co,r)g(r)  dwr 
JO 

+  I  f '  vc(r,  8,(x o),  Co,)gf\>-)C0.r  dw) . 
k**  l  Jo 


Therefore,  for  the  vectui  martingale  (S.3) 


Ps(x)C  =  E[C((^o.t(xq))C0,t]  + 


(8.4) 


Px(r,  t;lr(xo))g(r)  dwrC0,r 


+  I 

At”  I  J 


PrU;oAxo))g(k>(r)C0,r  dw). 
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Recall  that  D0tS  =  C~x,  so  forming  the  product  of  (6.4)  and  (8.4)  by  using  the  Ito  rule, 
we  have 


ps(x)  =  (pAx)C)D0,s 

—  £[ct(£u.T(-Vo))C’o.r]  — 


PritfAxoWr)  dr 


■  l 

k^l 


Pr(£oAx0))gfk\r)  dwkr  +  £  [  Pr(€oAx0))(g¥'(r))2  dr 

o  A.  =»  I  Jo 

Px{r,  ^O.r(-V0))g('  )  rtU’r+  Z  (  Pr(iioAXo))gtk\r)  d\Vkr 

U  I  Jo 


-  z 

k-  i  J 


!  pAr^lUo))g(r)gf(r)(lr-  £  f  pMlM)(gf(r))2  dr 
0  A.  =  l Jo 

=  £[c,(^r(.v0))C0,r]  -  I  pMtj  (x 0))f4(r)  dr 
Jo 

pAr,  6t(*o))g('')g^V)  dr. 


PAr,i*Axo))g(r)dwr-  Z 

0  A.  --I  J 


thus  establishing  the  result. 

This  verifies  by  a  simple,  direct  method  the  formula  of  Haussmann  [12]  without 
any  requirement  that  the  diffusion  coefficient  matrix  gg*  is  nonsingular.  However  we 
do  not  identify  p,(x)  with  the  gradient  of  the  minimum  cost  process;  this  follows  from 
arguments  as  in  [12]. 

9.  Conclusion,  Using  the  theory  of  stochastic  flows  the  ellect  of  a  perturbation  of 
an  optimal  control  is  explicitly  calculated  in  both  the  partially  observed  and  stochastic 
open  loop  cases.  The  only  difficulty  is  to  justify  the  differentiation.  The  adjoint  variable 
p,(x)  is  explicitly  identified. 

Theorem  9.1.  Iff  is  dijferentiable  in  the  control  variable  it,  and  if  the  random 
variable  x=£*,(xu)  has  a  conditional  density  q,(x)  under  the  measure  P*,  then  the 
inequality  of  Theorem  5.1  implies 

Z  (u,(x)~u'f(s))  I  I'U  .v)“(.v,  x,  u^qdx)  r/.vgO. 

j  I  Jkj  da, 

This  is  the  result  of  Bensoussan’s  paper  [1], 
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Abstract.  Using  stochastic  flows  a  minimum  principle  is  obtained  when  a  diffusion  is 
controlled  using  stochastic  open  loop  controls.  An  equation  for  the  adjoint  process  is  then 
derived  using  an  explicit  formula  for  the  integrand  in  a  certain  stochastic  integral. 


1.  Introduction. 

There  have  been  many  proofs  of  minimum  principles  in  stochastic  control.  For  a  small 
mple  see  the  works  of  Kushner  (15j,  Bismut  (2),  Haussmann  (10),  (11),  (12),  Davis  and 
Varaiya  (6),  and  the  book  by  Elliott  (8).  In  this  paper  we  consider  a  diffusion  and  stochastic 
open  loop  controls,  that  is,  controls  which  arc  adapted  to  the  filtration  of  the  driving 
Brownian  motion  process.  For  such  controls  the  dynamical  equations  have  strong  solutions, 
and  the  results  on  the  differentiability  of  the  solution,  due  originally  to  Blagovcscenskii  and 
Freidlin  (l),  can  be  applied.  The  work  of  Kunita  |14]  and  Bismut  (2)  on  stochastic  flows 
enables  the  variation  in  the  expected  cost,  due  to  a  perturbation  of  the  optimal  control,  to 
be  calculated  explicitly.  The  minimum  principle  follows  by  differentiating  this  quantity. 

If  the  optimal  control  is  Markov  the  stochastic  integral  representation  result  of  (9)  is 
applied  to  give  an  expression  for  a  quantity  associated  with  the  adjoint  process.  Stochastic 
calculus  is  then  used  to  derive  the  equation  satisfied  by  the  adjoint  process. 
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2.  Dynamics. 

Suppose  the  state  of  &  system  is  described  by  a  stochastic  differential  equation: 

Ail  =  f{t^t,v)dt  +  g(t,(t)dwt 

6  €  Rd,  to=x0,  0  <t<T.  (2.1) 

The  control  parameter  u  will  take  values  in  a  compact  subset  U  of  some  Euclidean  space  Rk . 
We  shall  make  the  following  assumptions. 

Ai:  I  :  (0,T)  x  Rd  xU  -  Rd  is  Borcl  measurable,  continuous  in  u  for  each  (<,z), 
continuously  differentiable  in  z  and  for  some  constant  K 

(1  +  W)"1  l/(<.*.«)|  +  !/*(<.*. «)i  <  Kx. 

Ay.  g:[0,T\xRd  -+  Rd®Rn  is  a  matrix  valued,  Borel  measurable  function,  continuously 
differentiable  in  z,  and  for  some  constant  Ki 


|j((f,x)| +  !</,(«, z)|  <Ki. 

The  columns  of  g  will  be  denoted  by  for  Jt  =  1, . . . ,  n. 

A3:  w  =  (to wn)  is  an  rt-dimensional  Brownian  motion  on  a  probability  space 
(fl ,F,P)  with  a  right  continuous,  complete  filtration  {/)},  0  <  t  <  T. 

DEFINITION  2.1.  The  set  of  admissible  controls  U_  will  be  the  Ft -predictable  functions  on 
(0,  rjxfi  with  values  in  U.  These  are  sometimes  called  ‘stochastic  open  loop’  controls,  [3j. 


REMARKS  2.2.  For  each  u  S  U  there  is,  therefore,  a  strong  solution  of  (2.1),  and 
we  shall  write  (z)  for  the  solution  trajectory  given  by 

Cj  (x)=x  +  I,  /(r,  £“r  (z),  uT)dr  +  J  g(r,  {z))dwr.  (2.2) 

Then,  because  u  is  a  (predictable)  parameter,  the  result  of  Blagovenscenskii  and  Frcidlin 

df* 

[ij  extends  to  this  situation,  sc  the  Jacobian  -^^-(z)  =  D*.  exists  and  is  the  solution  of 


Ki  ='+  [' f<(r,GAx)>"r)Dl 'dr+Y,  ['  gf  (r,C.r  (*)W,  dwf.  (2.3) 

■'*  *=I  ■'* 
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Here  /  is  the  d  X  d  identity  matrix.  In  fact,  if  the  coefficients  /  and  g  are  Ck  the  map 
is  Ck~l. 

Consider  the  matrix  valued  process  H  defined  by: 

Kt  f  Kr(fdr>CA*)>“r)-'t9(l)(r,Qr(X))')dr 

Jt  t=l 

(2-4) 

*=1 

Then  using  the  Ito  rule  we  see  d(//“(  D“( )  =  0  and  //,“£)“,  =  I,  so  7/“(  =  (£>“<  )-1 . 

Write  ||r(xo)||t  =  sup  |£o<(xo)l-  Then,  as  in  Lemma  2.1  of  [12|,  for  any  p, 

0<«<( 

1  <  p  <  oo,  using  Gronwall’s  and  Jensen’s  inequalities 

IKU (xo)l^  <  C(l  +  |z0|p  +  |  J‘  g(r ,  Hr  (xo))dwr|”) 

almost  surely  for  some  constant  C.  Therefore,  using  Burkholder’s  inequality  and  hypoth¬ 
esis  /1j,  ||£“(xo)||r  is  in  W  for  all  p,  1  <  p  <  oo.  Write 

P>“||r=  sup  \Dl\ 

o  <i<r 

||/7“||r=  sup  |//0“J. 

0  <i<T 

Then,  because  /(  and  g(  are  bounded,  an  application  of  Gronwall’s,  Jensen’s  and  Burk¬ 
holder’s  inequalities  again  implies 

JJ Z?“ Dx-  and  ||//“||r  are  in  Lp  for  all  p,  1  <  p  <  oo. 

COST  2.3.  Suppose  for  simplicity  that  the  cost  associated  with  the  process  is  purely 
terminal  and  given  by  a  bounded  CJ  function 

c((o,T  (zo))- 

A*:  We  suppose  |c(x)|  +  |cx(z)|  +  \czz  (z)|  <  K3(l  +  |m|<?)  for  some  q  <  oo. 
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The  expected  cost  if  a  control  u  €  U  is  used  is,  therefore, 


J(u)  =  £[CUoV(xo))|. 


We  shall  suppose  there  is  an  optimal  control  u*  G  U  so 


J(u*)<J(u)  for  all  uGU. 

NOTATION  2.4.  If  u*  is  an  optimal  control  write  f*  for  (u  ,  D*  for  D*  etc. 
REMARKS  2.5.  Consider  a  d-dimensional  semimartingale  of  the  form 

2j  =  a,  +  At 

where  A  is  a  predictable  bounded  variation  process.  Then  Kunita’s  formula  ( 14]  for  the 
composition  of  processes  can  be  applied,  (see  also  Bismut  [5]),  and  we  have 

£.*,«(*<)  ~z,+  jt  f(r>C.,izr),K)dr 

+  I,  ~JT^dAr  +  J^  j' 


(2.5) 


DEFINITION  2.6.  Consider  perturbations  of  the  optimal  control  u*  of  the  following  kind: 
For  s  €  (0,  T\,  h  >  0  such  that  0<s<s  +  h<T,  and  A  6  F,  define,  for  any  other 
admissible  control  uG.lL, 

(,  \  f  “*(*>“')  £  Is,5  +  /‘l  x  A 

Ull'Wl  =  < 

(  u(t,u;)  if{t,w)  G  (s,s  +  A)  x  A. 

Applying  (2.5)  we  have,  similarly  to  Theorem  5.1  of  |4j,  the  following  result. 
THEOREM  2.7,  For  the  perturbation  u  of  u'  consider  the  process 


Z‘=I  +  /  (d'~fx~~')  (f(r’Zlr(zr),ur)~  f(r,{',r(zr),u'r))dr. 


(2.6) 


Then  the  process  C,i  izt)  >s  indistinguishable  from  (x). 

PROOF.  Substituting  (2.6)  in  (2.5)  we  see 

C,tizl)  =  x  +  J  f(T'  {*.'  (zr)>ur)dr 

+  J'  9{r>C,r{zr))dwr 

=  X  +  J'  /(f,  fjir(^r),Ur)£tr  +  ^  9{r,V,,r[zr))dmf 

However,  the  solution  to  (2.2)  is  unique  so  (zr)  =  £<,(  (z)- 

REMARKS  2.8.  Note  that  u(t)  =  u'(/)  if  t  >  s  +  h  so  z(  =  zt+h  if  t  >  s  +  h. 


Therefore 


£,<(*<)  =  C(**+*)  =  (&+*  (*)) 


Dccau: 


if  t  >  s  +  h. 


This  gi 
in  the  < 
to  justi 
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3.  A  Minimum  Principle. 


J(u)  =  £|c(£o,r(xo))] 

=  ( x )))  where  x  =  £<>,*  (xo), 

because,  by  uniqueness,  fo,T  (xo)  —  C.T  (•*■)•  Similarly, 

J(u)  =  £|c(^r(i0))| 

=  £|c(e.vw)i 

=  E\cie,x  (*•+>>))]■ 


Therefore, 


J(u)  -  J(u’)  =  E{c(t‘,j  ( z.+h  ))  -  c(C,t  (*)))• 


Because  C,t  (')  *s  differentiable  this  is 

=  E[f^  c({V,iT(zr))^~{^^yl  (f(r,C.A*r),«r)-f{r,C.A*r)X))dr}- 

(3.1) 

This  gives  an  explicit  formula  for  the  change  in  the  cost  resulting  from  a  ‘strong’  variation 
in  the  optimal  control.  It  involves  only  a  time  integration.  The  only  remaining  problem  is 
to  justify  the  differentiation  of  the  right  hand  side  of  (3.1). 

Write  = 


J(u)  -  J(u*)  =  ^  £[(r(s,r,zr)  -  r(s,r,x))(/(r,  ^r(2r),  ur)-/(r,  £jif(ar),  u;))jdr 

f  J  i?[(r(4,r,x)  ~  r(r,r,z))(/(r,  (;r(zr),  «r)  -f(r,  c,r(zr),  K))]dr 

+  I.  -E[r(r'r>I)^(r’  C,r(*r).  Ur)-/(r,  £,r(*r),  <) 

-f(r,  C.,r  (x),  Ur)  +  f(r,  £r  (z),  Ur*))]</r 
+  y  ^[r(r,r,j:)(/(r,  ^  (x0),  ur)  -  /(r,  (i0),  «;))]* 

=  /,(/1)  +  /j(/I)  +  /3(A)  +  /<(/1),  say. 
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Now, 


|/,(/i)|  <  Kt  J  fc’[|r(s, r,zr)  -  r(s,r,x)|(l  +  |Ku(zo)||.+*  )] dr 

<  KAh  sup  £:f|r(4,r,*r)  -  r(s,r,z)|(l  +  |U“(r0)|U*)| 

i<r<t+h  1  1 

iM^oi  ^  Ks  J  ^(ir('s.r»z)  -  r(r.r,*)i(i  +  iir(^o)iu)]* 

<  Ksh  sup  £[|r(s,r,zr)  —  r(r,r,z)|(l  +  ||£“(zo)||<+/i  1 

i<r<j+A  I-  J 

IMA)I  ^  Ka  J  ^ [lr(r»  r» x)l  II1  -  ZA\ ]* 

<  K6h  sup  £[|r(r,r,x)|  ||x  -  z.||,+J. 

4<r<4+A  *•  * 

The  differences  |r(s,r,xr)-r(s,r,x)|,  |r(s,r,x)-r(r,r,x)|  and  ||x-z||,+a  arc  all  uniformly 
bounded  in  some  W ,  p  >  1,  and 

iim  |r(s,r,zr)  -  r(s,r,z)|  =  0  a.s. 
iim  |r(s,r,z)  -  r(r,r,z)j  =  0  a.s. 


Iim  |jx  -  z. ||,+a  =  0. 

A-*0 


Therefore, 


jim  ||r(s, r,zr)  —  r(«,r,z)||f,  =0 

jim  ||r(s, r,x)  -  r(r,r,x)||,  =  0 

and  Iim  ||(!jz  -  z||,+fc  )j|p  =  0  for  some  p. 
h— 0 

Consequently,  lim  h~l  /*(h)  =  0,  for  k  =  1,2,3. 
h— *0 

The  only  remaining  problem  concerns  the  differentiability  of 

/4(h)  =  J  E[r(r,r,z)(/(r,  ^o,f  (^o),  «r)  -  /(r,  £o,r(zo),  u;))jdr. 

The  integrand  is  almost  surely  in  Z/'((0,!rj)  so  Iim  h~l  /<(/«)  exists  for  almost  every  s  € 

A— *0 

(0,T).  However,  the  set  of  times  {s}  where  the  limit  may  not  exist  might  depend  on  the 


cor 

per 


lim 
A— 0 


Consc 
limit  i 
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control  u.  Consequently  we  must  restrict  the  perturbations  u  of  the  optimal  control  u*  to 
perturbations  from  a  countable  dense  set  of  controls.  In  fact: 

1)  Because  the  trajectories  are,  almost  surely,  continuous,  F p  is  countably  generated 
by  sets  {,4,,},  »  =  1,2,...  for  any  rational  number  p  G  (0,T|.  Consequently  Ft  is 
countably  generated  by  the  sets  {A^},  r  <  t. 

2)  Let  Gt  denote  the  set  cf  measurable  functions  from  (fl ,Ft)  to  U  C  Rk.  (If  u  €  £ 
then  u(l,w)  G  Gt-)  Using  the  L'-norm,  as  in  j7),  there  is  a  countable  dense  subset 
// f  =  {u„}  of  G,,  for  rational  p  G  (0,T|.  If  lit  —  (J  Hp  then  13  4  countable 

p<t 


dense  subset  of  Gt.  If  G  llf  then,  as  a  function  constant  in  time,  Ujf  can  be 

considered  as  an  admissible  control  over  any  time  interval  (f,  r|  for  t  >  p. 

3)  The  countable  family  of  perturbations  is  obtained  by  considering  sets  A{f  G  Ft, 

functions  p  6  lh,  where  p  <  f,  and  defining  as  in  3.1 

f«'M  if  (s,u>)  ^  (t,r|  X /t,> 
lf  S'W  1  if  («,*»}  G  \t,T\  x  Aip. 


Then  for  each  i,j,  p 

K™h~l  j  £'[I'(r.r.x)(/(r.  £o,r(xo),  u‘p)~f(r,  £o,r(xo).  «'))]dr  (3.2) 

exists  and  equals 


£[r(s,s,x)(/(s,  ^(xo),  «/,)-/(*  (o,A*o),  «*))/*,] 
for  almost  all  s  G  (0,  T). 

Therefore,  considering  this  perturbation  we  have 

I1.™  r‘(</(u//)  “  J(u‘))  =  ^[r(«,«,*)(/(3.  £o,,(xo).  J{*,  to,Ax o).  «*))U,] 

>0  for  almost  all  s  G  (0,T). 


Consequently  there  is  a  set  5  C  (0,Tj  of  rcro  Lcbesgue  measure  such  that,  if  a  $  S,  the 
limit  in  (3.2)  exists  for  all  »’,/, p,  and  gives 

£[r(a,s,x)(/(s,  £o,,(xo).  «;p)  ~  f(s,  io.Axo),  u*))U,]  >0. 
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Using  the  monotone  class  theorem,  and  approximating  an  arbitrary  admissible  control 
u  €  U  we  can  deduce  that  if  s  $  S 

£i|r(s,s,x)(/(s,  £o,f  (x0),  u)  -  /(s,  £o.*(xo),  >  0  for  any  u  <E  U  and  A  G  ft. 

(3.3) 

Write 

P«(x)  =  I  F‘]  =  ^[r(5,5,x)  |  ft)  (3,1) 

where,  as  before,  x  =  £o,#(xo)-  Then  p,(x)  is  the  adjoint  variable  and  we  have  in  (3.3) 
proved  the  following  minimum  principle: 

THEOREM  5.1.  If  u‘  6  U_  is  an  optimal  control  there  is  a  set  S  C  (0,  T|  of  zero  Lebesgue 
measure  such  that  if  s  <£  S 

p1(x)/(s,x,u')  <  p,(x)f(s,x,u)  a.s. 

That  is,  the  optimal  control  u‘  almost  surely  minimizes  the  Hamiltonian  and  the  adjoint 
variable  is  p,  (x). 

REMARKS  3.2.  Under  certain  conditions  the  minimum  cost  attainable  under  the 
stochastic  open  loop  controls  is  equal  to  the  minimum  cost  attainable  under  the  Markov, 
feedback  controls  of  the  form  u($,  (xo)).  See  for  example  |2],  [lOj.  If  is  a  Markov 

control,  with  a  corresponding,  possibly  weak,  solution  trajectory  fUA/ ,  then  u\{  can  be 
considered  as  a  stochastic  open  loop  control  uat(iu)  by  putting 

uM(w)  -  uM(s,  $“{x0,w)). 

This  means  the  control  in  effect  ‘follows’  its  original  trajectory  than  any  new  trajectory. 
That  is  the  control  is  similar  to  the  adjoint  strategics  considered  by  Krylov  (l3j.  The 
significance  of  this  is  that  when  we  consider  variations  in  the  state  trajectory  ft  and 
derivatives  of  the  map  x  — »  ft_<  (x),  the  control  does  not  react,  and  so  we  do  not  introduce 
derivatives  in  the  u  variable. 


4.  T 

consi< 

highc 

Thec 


r 

ness  of 

so,  di(T< 

where  l 
Ft 

so  from 


If  the  optimal  control  u’  is  Markov  the  process  is  Markov  and 

p,(r)  =  E[r{s,s,x)  |  F,) 

=  ^(r(s,s,2)  |  x\. 


4.  The  Adjoint  Process. 

Suppose  the  optimal  control  u'  is  Markov.  As  noted  above,  u‘  can  and  will  be 

.  dCr  .  ,  /dC.T\-1 

considered  as  an  open  loop  control.  The  Jacobian  — exists,  as  docs  I  ~ff~-  j  and 
higher  derivatives. 

THEOREM  4.1.  Suppose  the  optima]  control  u*  is  Markov.  Then 

P,[x)  =  £|c<Uo,r(*o))A>,r)-  /  Pr(to,r  (xo))h{r,to,,  (*o)X)<fr 

Jo 

+  /  Pr,(r'to,r(xo))g[rt(o,r(xo))dWr 

Jo 

~  I  Pz(r,(o.r(xo))g(r,to.r{xo))9((r,toir(xo))dr. 

Jo 

PROOF.  Write  f((r)  for  o.r(*o),u?)  and  g(r)  for  g(r,  (io)),  etc.  By  unique¬ 
ness  of  the  solutions  to  (2.1) 

V ,T  (xo)  =  V.,t  {Vo,.  (xo))  (4.1) 


so,  differentiating, 


Dq.T  =  D,j  Do,, 


where  Dq,t  —  D'0T  etc.  (without  the  ’). 
From  (3.4)  and  (3.5) 


so  from  (4.2) 


p,(x)  =  EIc(((qT(xo))D,,t  |  -F,! 


P.{x)Do,.  =  (x0))D0,t  |  F,j 


li 
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and  this  is  a  (P,  {.F<})  martingale.  Write  x  =  (xo),  D  =  jDo,,.  From  the  martingale 

representation  result  (9),  the  integrand  in  the  representation  of  p,(x)D  as  a  atochastic 
integral  is  obtained  by  the  Ito  rule,  noting  that  only  the  stochastic  integral  terms  will 
appear.  These  involve  the  dcrivaliv  s  in  x  and  D.  Therefore 

P,{x)D  =  £|c<(£o,r(*o))A>lr!  +  f  Pi{r,&lT{xo))g(r)dwrD0ir 

Jo 

+  X^/  P'((0,r(x0))9(l)(r)V0,rdw?.  (4.4) 

Jk=l  Jo 

Recall  from  (2.4)  that  Hq>(  =  D~l  so  forming  the  product  of  (2.4)  and  (4.4),  using  the  Ito 


Pt(x)  =  {p.{x)D)II0i, 

=  ^[c<(^0.7'  (a;o))-Do,7'  j  -  /  Pr(^0,r(z0))/<(r)^r 

Jo 

f  P'(£o,r  l1©))^  [r)dw$  +  ^  /  Mfo.r  (*o))(ff^*J 1  (r))3* 

t=l  jb=i  Jo 

+  f  Px(r,C0,r{xo))g{r)dwr  +  ^2  f  pr(C0if(x0))g[k)  {r)dw^ 

J 0  J 0 

-J2  f  Pl{r>to,r(xo))9{r)g(k){r)dr-Y,  f  Pr(£o,r  (io))^  (r))3* 

*=t  J°  k= 1 

=  £lc<UoJ  ixo))D0,T  I  -  /  Pr(Vo,T  (xo))/<(r)dr 
Jo 

r2  n  1*1 

+  j ^  pz[r,^  r(x0)}g(r)dwT  -  ^  pz(r,  (i0))j(r)(7^  (r)dr 
so  establishing  the  result. 

This  verifies  by  a  simple,  direct  method  the  formula  of  Haussman  ( 10] . 
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Solutions  of  Kolmogorov's  forward  and  backward  equations  arc  obtained  by 
considering  a  family  of  conditional  expectations  and  the  use  of  stochastic  flows  to 
justify  differentiation  in  the  time  variable.  !.  1989  Academic  Press,  Inc, 


Introduction 

Probabilistic  solutions  of  the  Cauchy  problem  for  Kolmogorov’s  forward 
and  backward  equations  have  been  known  for  many  years.  In  [5, 6] 
Kunita  uses  stochastic  flows  associated  with  forward  and  backward 
stochastic  differential  equations  to  write  down  explicit  forms  of  the  solu¬ 
tions.  In  fact  he  uses  both  forms  simultaneously,  in  that  he  requires  the 
backward  equation  for  the  forward  process  to  solve  the  Kolmogorov  back¬ 
ward  equation,  and  the  forward  equation  for  the  backward  process  to  solve 
the  Kolmogorov  forward  equation.  In  this  note  we  indicate  how  solutions 
of  the  Kolmogorov  equations  can  be  obtained  directly  by  differentiation  in 
the  time  variable  of  a  family  of  conditional  expectations.  This  is  justified  by 
differentiating  inside  the  conditional  expectation,  using  the  properties  of 
stochastic  flows.  Both  the  forward  and  backward  Kolmogorov  equations 
are  considered.  As  noted  in  [3],  the  conditional  expectation  is  a  solution 
of  Kolmogorov’s  equation  because  the  bounded  variation  term  in  its  semi- 
martingale  representation,  using  the  Ito  formula,  is  zero. 
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1.  Stochastic  Flows 

Let  w,  =  (if),  be  m-dimensional  standard  Brownian  motion 

defined  for  all  l^Oona  fixed  probability  space  (Q,  F,  P).  Write  (F,),^ 0  for 
the  right-continuous  complete  filtration  generated  by  if. 

Suppose  that  we  are  given  vector  fields  X0,  Xt, Xm  on  [0,  oo )xRJ 
such  that  each  X,  is  three  times  continuously  differentiable  with  all 
derivatives  bounded.  We  can  associate  a  stochastic  flow  with  if  and  the  X,: 


Theorem  l.l  [2].  There  is  a  map  c,:  [0,  co)  x  [0,  co)  x  W‘  xQ  U 
such  that 

(i)  for  O^s^t  and  xeW,  <;„(-v)  is  the  essentially  unique  solution  of 
the  stochastic  differential  equation 

dcsJ.\)  =  X0(t,  ffx))  dt  +  X,(t,  ZsJx))  dw\  (1.1) 

with  Cj.5(.v)  =  .y  {we  employ  the  Einstein  summation  convention  when  con¬ 
venient); 

(ii)  for  each  s  and  t  and  almost  all  if,  the  map  £,.,(■)  has  a  version 
which  is  twice  differentiable  in  x  and  continuous  in  s  and  t.  The  Jacobian 
Ds,  =  dqSJ/dx  is  then  a  solution  of  the  linearized  equation 


=  ~  (t,  ?,»)  DSJ  dt  +  ~  (t,  Cjx))  D,u  dw\; 


(iii)  the  second  derivative  d2c^,/cx2  is  well-defined  for  each  x,  s,  t. 

Remark.  The  conditions  we  have  imposed  on  the  coefficients  of  (1.1) 
suffice  to  ensure  that  qSJ  and  DSt,  belong  to  Lr(Q)  for  all  p>  0.  The 
arguments  to  establish  appropriate  estimates  are  standard  applications  of 
the  Burkholder,  Jensen,  and  Gronwall  inequalities,  similar  to  those  in 
[1,4].  The  conditions  on  the  coefficients  can  be  relaxed  somewhat:  those 
employed  here  are  fixed  in  the  interests  of  simplicity. 

Now  restrict  attention  for  the  remainder  of  this  paper  to  the  time 
interval  [0,  T ],  where  T  is  a  fixed  finite  positive  number.  Consider  the 
trajectories  with  a  fixed  initial  position  .VoeR''.  and  assume  that  we  are 
given  a  function  c(?0.r(.Y0))  of  the  final  position  at  t  =  T.  Assume  further 
that  c  is  three  times  continuously  differentiable  with  bounded  derivatives. 

Define  the  (F,)-martingale  M  =  T  by  setting 


M ,  —  F[c(c0  r(.Y0))|F,]. 

It  is  shown  in  [3]  that  the  uniqueness  of  the  martingale  representation 
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>  M,  =  M0  +  §‘0y,(s)  dlV‘s,  where  the  integrands  y,(j)  are  predictable  pro- 
‘  cesses,  can  be  exploited  to  find  an  explicit  expression  for  y,(.v)  and  to  solve 
;  the  Cauchy  problem:  in  fact,  if  we  set  .v  =  £0./(-vo)  and  write  (as  in  Section  2 
i  below)  M,  =  =  V(t,  x),  then  it  is  shown  that  V  is  a  solution 

of  the  backward  parabolic  differential  equation  dV/dt  +  LV=  0,  with  final 
condition  V(T,  ur)  =  c(xT),  where  L  is  the  operator 


d  pi  i  d 


Lj*=l 


m  /  S2 


In  [3]  the  Ito  differention  rule  is  applied  to  K(t,).Y)  to  yield  these  results. 
This  requires  the  differentiability  of  V  in  i,  which  can  be  proved  using 
Kunita’s  results  [6]  on  the  reverse  time  stochastic  differential  equation  for 
the  reverse  flow  <:*.,(•)■  Our  purpose  in  this  note  is  to  show  that  this  can 
be  avoided  by  a  direct  calculation  of  dV/dt.  This  approach  has  the  advan¬ 
tage  that  only  the  semigroup  properties  of  the  flow  c,s„  the  Markov 
property,  and  the  independence  of  Brownian  increments  are  required  for 
the  proof. 


2.  Differentiability 

Because  ir0  =  0  for  standard  /n-dimensional  Brownian  motion  we  observe 
that 

•  F,  =  cr{wi:  /} 

=  Ff  -  <r{  w,  -  w, :  0  ^  r  ^  .v  sc  t }. 

f  More  generally  we  shall  write  Fl  =  a{ws-wr:u^s^t}.  Therefore,  if  the 
i  initial  condition  .v0  of  the  trajectory  is  known,  by  the  Markov  property  and 
t  independence  of  future  increments 

|  £[c(Co,r(.v0))|/:-,]  =  ^[c(«o.r(.Vo))l^0] 

I  =£[c(?o.t(.v0))1.y] 

f£-  —  ^(..vCc(Co,r(-vo))3 

1:  =  V(t,  -v),  where  x  =  c0  ,(.v0). 

%  Now  by  the  uniqueness  of  solutions  of  (1.1),  and  the  semigroup 
J§  property, 

Jf 

gj  4o,t(xo)  —  ^i.t(^0,i(xo)) 

J§  =C,  7(.v). 
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Therefore, 


Write 


K(/,.v)  =  £,,[c(c,7(.v))] 


(2.1) 


D,.,  dx'  (vj. 


H',,= 


d%, 

dx; 


■M 


Differentiating  (2.1)  in  x  we  see 


—  (/,  x)  =  VJl,  x)  =  £[c{(c,T(.v))  DuT | £,] 

0  (/,  .v)  =  .y)  =  £[c..(c,7(.y))  Z),r(g)  /),.;•] 


+  [^«(c;,r(-v)) 

because  the  map  ,v -» |(>T(,v)  is  C2. 

We  now  wish  to  investigate  the  differentiability  of  V(t,  x)  in  the  t 
variable.  Under  strong  enough  conditions  on  the  coefficients  X0,  X,  this 
would  follow  as  in  [6]  by  considering  the  reverse  time  stochastic  differen¬ 
tial  equation  for  the  reverse  flow  £,,(  ■ ).  However,  we  will  give  a  direct 
proof. 


Theorem  2.1.  For  any  xe  R‘‘,  V(l,  x)  is  continuously  differentiable  in  t 
and 


dV  ‘‘  T  8c 

-  77  ('>  *)  =  I  m  -V)  E  |  c^r.Mx))  ~^j 


1  (I  lit 

+  7  I  X  J?(/,  -V)  X‘(t,  x)  £  [a,  a;(c(c,,r(.v))3 

1  k,l=\  Jn  i 


=  LV(t,  x),  where  dk  =  d/dxk. 
Proof.  For  a  given  xs  R'1  consider 

•  \*»  —  ^LCIS/. 


!,W:  f'K' 
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Using  the  mean  value  theorem,  for  h  >  0, 
c(C;-A.r(-v))-c(c,.r(A-)) 

=  c(£,.r(C, -/,.,(*)))  -  c(q,Mx)) 

=  c<(c,.7(-v))  -  x) 

1  d 

+  5  I  Ok  ')(cT-„.,(-v)-a"'), 

L  kj-  1 

where  Ic.-a.X-v)  —  a)1  and  x~  (*«, x‘‘),  where  xeRf. 

Now 


C/-/,.,(-v)  =  x  +  (  X0(r,  c,_/I>r(.v))  dr 

+  j  X,(r,  $t-h.r(x))dw,r, 

so  (Oi-h.i(x)  -  x)  is  F'~h  measurable.  For  the  given  .v,  c,.r(.v)  and  5qhT/dx 
arc  F‘r  measurable,  so  because  the  increments  of  Brownian  motion  are 
independent 

=  E  [<*(«, i7(f,.  r(-v)))  E  |^f  ^  X0(r,  f ,  _  ,,r(.v))  rfr] 

=  o(h).  (2.2) 


Write 


t  dk  oMi,,A  •  )))('/)(?L/, .,(•'■)  -  .v')(?7-/,t(.v)  -  -v'") 

*./=.  i 

=  Z  (^k^l{c(ii. ?( ’)))(>/) 

k.t-  1 

-  Ok  d,(c(4,.A  ■  )))(*))(£'_*,(*)  -  x'Xc'/L.Jx)  -  xm) 

+  Z  ^^(c(7,r(-)))(-v)(cZA.,(-v)--v') 

i 

xW-lJx)-x'")  =  Sl  +  S2, 


say. 
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Because  the  second  derivatives  of  c  are  bounded,  and  because  c  and  D 
belong  to  U’(Q)  for  all  p, 

!  r (d2  d2  \2l\i/2 

■(£CiC,-/,,(v)-.v|4]1/2) 

/  r id2  c2 

<£:(£  (^2(co?,.r)0/)-T^  U'oC,.r)(.v)j  J  h 

—  o{li)  (2.3) 

because,  as  /;-» 0,  (d2/dx2)(c°£lmT)(>i)-(d2/dx2)(cog,.T)(x),  converges  to 
zero  a.s.  and  is  bounded.  Again,  because  F‘~h  and  F^arc  independent 

£[s2]  =  e  £C<5^,(c»c,.r)(-v)]  ^C(^^,1(-v)--v/)(c:;"_/,,(.v)-.v"')] 

A,/=>  1 

«  E  £CM/(^,r)(-v)] 

k,t  *  1 

X  E  £[*,■('•>  f, _*.,(*))  X‘M-  ?-/„,)]  rfr 

y»l 

= «(//).  (2.4) 

Front  (2.2),  (2.3),  and  (2.4)  dividing  by  h>0  we  have 

lint /;  l(V(t  —  h,x)~V(t,x)) 

/;-0 

=  ,Y0(/..V)£  c{(«, ./■(*)) 

|  d  m 

+-,  E  E  */('.  -v)  -v)  £[s*  fv(c  'c,t)(a-)]. 

A'./=  I  j  =3  I 

This  establishes  the  existence  of  the  left-hand  derivative  of  V  at  t.  The 
existence  of  the  right-hand  derivative  is  proved  by  considering 

lr'(V(i,x)~V(t  +  h,.x)) 

=  Ir'iEMS.'Ax))  3  -  £[c(c,+/,r(.v))]) 

=  //-1(£[c(c,+/,r(c,,  „(*)))] -£lctfl+h,Ax)m 

Using  the  mean  value  theorem  the  limit  as  /;  -» 0  is  established  as  before, 
introducing  terms  similar  to  5',. 


i 


32 


ELLIOTT  AND  KOPP 


Therefore,  —  (dV/dt)(t,  x)  exists  and  equals  LV(t,  x). 
Corollary  2.2.  Recalling  the  operator 


we  see  that  V(t,  x)  =  £[t'(<0.r(A'o))i  Z7,0]  is  the  solution  of  the  backward 
Kolmogorov  equation 


Jt 


+  LV=  0 


with  terminal  condition 


V(T,xr)  =  ELc(<;0'T(x0mF0r-} 

=  C(.Yr). 


1- 

P 

zf 

f- 

g 

f 

V 

£ 

f- 


lUi 


Remark  2.3.  The  situation  when  the  coefficients  T,  are  not  bounded  can 
be  treated  by  using  stopping  times.  Equations  of  the  form 

^■+‘LV+<fiv  =  0, 
at 

where  <t>  is  a  smooth,  bounded  function,  can  be  treated  by  introducing  a 
new  coordinate  c',+ ',  where 

H7  '(*»  y)  =  3'  exp  j  j  <t>(U  eM(x))  r/rj 

=  3’+f  fsf  1  <!>(>■,  £„(*))  dr. 

J5 

This  is  the  Feynman-Kac  form  of  the  solution;  see  Kunita  [5], 

3.  The  Forward  Equation 

We  again  work  on  the  time  interval  [0,  T].  Recall  the  cr-field 
F‘r  =  a{ws -  wr :  /  <  r  ^  s  <  T}, 

and  denote  by  {F'T},  O^t^T,  the  left-continuous  complete  filtration 


lll!'kJ.lJ  lliPillHUI 1  Ml 
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generated  by  the  F'r.  Suppose  /(/•)  is  a  continuous  {F'r}-adapted  process, 
that  is,  f(r)  is  Fr  measurable.  Then  the  Ito  backward  integral 

f  f(r)3wr 


is  defined  as  the  limit  in  probability  of  sums 

*  =  o 

where  /  =  ^  T)  and  the  limit  is  taken  as  |d|  = 

max  |tA.+  ,  —  Z*i  goes  to  zero.  Kunita  [5,6]  then  defines  the  backward 
stochastic  differential  equation 

dL  =  -  Ux,  <f s)  ds  -  Xj(s,  is)  Aw1, . 

For  each  te[0,  T]  and  xeR‘'  this  has  a  solution  <f,.r(.v)  with  terminal 
condition  ^ri7(x)  =  .v.  We  shall  suppose  the  coefficient  vector  fields  Xh 
i  =  0, ...,  m,  satisfy  the  boundedness  and  smoothness  conditions  of  Section  2. 
In  fact  <f,.r(.v)  is  given  by 

<f ,.AX)  ~  x  +  (  xo(n  <f,. 7 (•'•"I)  dr  +  |  Xj(r,  <fr.r(.v))  ciw[.  (3.1 ) 

Ji 

Clearly  is  a  backward  { F'r }  semimartingale,  with  the  time 

parameter  i  running  from  T  to  0.  The  situation  is  the  mirror  image  to  that 
discussed  in  Sections  2  and  3,  so  there  exists  a  map 

<f:  [0,  r]x[0.  TlxR-'xQ  ->/?■', 


such  that  for  ?</-<  T  and  xsRd,  ir,T(x)  is  the  essentially  unique  solution 
of  (3.1).  Furthermore,  i  is  twice  differentiable  in  a-;  we  shall  write 


D,r  for 


For  a  given  terminal  condition  xT  consider  the  backward  solution  £0.r(*r) 
and  the  quantity  c(|0.7-('v/-))>  where  c  is  a  bounded,  C3  function  on  Rd,  with 
bounded  derivatives.  We  now  consider  the  backward  martingale 


M,  —  F[.c{$qj(xt))  |  F‘r2- 
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By  the  Markov  property  this  is  equal  to 

£[c(?o.r(-vr))!*v]>  wliere  x  =  t.Axr) 

=  E[c(lo., (*))!*] 

=  V[i,  x),  say. 

Because  of  the  symmetrical  nature  of  both  the  Markov  property  and  the 
independence  of  Brownian  increments,  the  analog  of  the  argument  in 
Section  2  for  the  forward  flow  applies  to  V{t,  x)  and  shows  that 

8V 

—  (t,x)  =  LV(t,x). 

U'Jote  that  because  time  is  now  considered  in  a  negative  direction  we 
consider  c(|0.,+*(.v))  — c(lo,i(-v))  and  so  we  obtain  dV/dt,  rather  than  the 
-dV/dt  of  Theorem  2.1.) 

The  initial  condition  for  V  is  1^(0,  i0  T(xr))  =  c(^T(xT)),  i.e., 
V(Q,  A'0)  =  c(.y0). 

This  discussion  avoids  the  “forward  equation”  for  the  “backward 
process,”  as  used  by  Kunita  [5], 
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An  integration  by  parts  formula  for  functions  of  jump  process  is  established  which  follows  from  an 
ordinary  integration  by  parts  in  the  stale  space  of  the  jump  measure.  The  analog  of  the  Malliavin 
matrix  is  defined;  if.the  inverse  of  this  matrix  belongs  to  all  I,  the  jump  process  has  a  smooth 

density. 

KEY  WORDS:  Malliavin  calculus,  jump  processes,  integration  by  parts,  stochastic  Rows. 


0.  INTRODUCTION 

The  Malliavin  calculus  is  a  calculus  of  variations  in  function  space.  One  of  its 
principal  applications  is  to  establish  the  existence  and  smoothness  of  densities  of 
processes  defined  by  stochastic  differential  equations.  Following  the  original  work 
by  Malliavin  [10],  Bismut  [4],  Stroock  [14],  and  others,  on  continuous  stochastic 
differential  equations  driven  by  Brownian  motion,  there  have  been  papers  on  the 
Malliavin  calculus  for  equations  driven  by  jump  processes.  See,  for  example,  the 
work  of  Bismut  [5],  Bichteler  and  Jacod  [2],  Bichteler,  Gravereaux  and  Jacod  [3], 
Leandre  [9],  Bass  and  Cranston  [I]  and  the  recent  paper  by  Norris  [13],  A 
central  result  in  these  papers  is  an  integration-by-parts  formula  in  function  space 
which  is  often  established  by  considering  a  perturbation  of  the  original  process 
and  a  Girsanov  change  of  measure.  However,  we  show  below  how  an  integration- 
by-parts  formula  can  simply  be  obtained  from  classical  integration  by  parts  in  the 
state  space  of  the  jump  measure.  An  analogous  integration-by-parts  formula  for 
continuous  diffusions  is  established  in  [6];  however,  the  jump  processes  discussed 
in  this  paper  require  significantly  different  definitions  and  techniques.  A  Malliavin 
matrix  M  is  introduced  and  if  M'1  belongs  to  all  the  spaces  Lp,p2. 1,  then  the 
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AFOSR-86-0332. 
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jump  process  has  a  smooth  density.  Of  course  the  crucial  problem  then  concerns 
the  integrability  of  A/*1;  this  is  investigated  in  [3]  and  [5]  by  imposing  growth 
conditions  on  the  coefficients  and  using  delicate  Tauberian  theorems.  However,  we 
do  not  discuss  such  problems  in  this  paper. 


I.  ASSUMPTIONS  AND  DYNAMICS 

Consider  a  probability  space  (Q.F.P).  Suppose  ft =/<(<«, dt  xdz)  is  a  Poisson 
random  measure  on  [0.7]  xZ  and  let  {F,},0gtgT  be  the  right  continuous 
complete  filtration  generated  by  ft.  Here  Z  can  be  an  open  unbounded  subject  of 
Rm:  in  fact  for  simplicity  we  shall  assume  Z  =  Rm.  G(d:)  =  dz  will  denote  Lebesgue 
measure  on  Z.  We  shall  suppose  the  compensator  v=v(dt  x  dz)  of  ft  is  of  the  form 
dt  x  G(dz)  =  dt  xdz,  so  that  the  compensated  measure  / i  of  ft  is: 
fi(dt  x  dz)  =  fdtll  x  dz)  —  dt  x  dz. 


Dynamics  1. 1 

Consider  a  jump  process  < ;0tl(x0)6  Rd,  for  OgrgT  given  by 

t 

vo..(*o)  =  *o  +  f  J  £(£o.,  -(*o),  =)i‘(dr  X  dz) 

07. 


+  J  J /i(<;'o.f*(Xo),:)v(drX(/r)  +  J  A(c0>r.(.x0))</r.  (1.1) 

0  7  0 

Here 

g  =  (g‘),  I  £i^d,  maps  Rd  x  Z-*Rd 
/i =(/]'),  I  gigd,  maps  Rd  xZ~Rd 
A-(A'),  \£i£d,  maps  Rd-+Rd. 

We  shall  assume  g,  h,  and  A  are  C*  and  have  bounded  derivatives  in  .x  and  z  of 
all  orders.  Furthermore,  we  suppose 


and 


sup \DpxDih{x.  •)  6  L'(Z)  n  L’(Z)  n  L»(Z) 
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for  p5  I,  and  that 


infdct  j/  +jf,(.v.-)|>0.  (1.2) 


Then  from  the  results  of  Meyer  [II]  and  Lcandre  [8]  there  exists  a  map 

</>:nx[0.r]x/C'-KJ 


such  that 

i)  for  each  xeR11  4>{io,t,x)  is  the  unique  solution  of  (1,1). 

ii)  for  each  w  and  t  the  map  </>((«, f,-)  is  Cx  on  RJ ,  with  derivatives  of  all  orders 
which  satisfy  the  stochastic  equations  obtained  from  (1.1)  by  formal 
differentiation. 

Consequently,  for  example, 

d<i>((o,l,x)  <?Co,,(.v) 


satisfies 


‘IDo.,  - sj|  Do.,  -  dp  +  vrz  D0.,  dv  +  ^D0i,.  dt 


with  D0t0  =  /.  the  d  x  d  identity  matrix. 

The  following  result  is  similar  to  that  of  Lcandre  [8], 

Lemma  1.2  Suppose  V  is  the  matrix  solution  of  the  stochastic  equation 


1  o.<  =  /  ~  J  J  vo.r  -  %  dp  -  j  J  V0.r-  fl  dv 

0  7  ch  0  7. 


(L3) 


d4  ' 

Jtdr  +  HVo,- 

1 S  0  2 


(1.4) 


Then  K0.,  =  Do.,‘  for  0 g/gT 

Proof  An  application  of  the  product  rule  shows  that  d(V0  ,D0  ,)  =  0,  so  because 
K).o^o,o  =  f  the  result  follows. 

Remarks  1.3  First  note  that  if  DSu,Wsu  are  solutions  of  (1.3)  and  (1.4)  with 
Dt,s  =  l  =  j,  then  by  applying  Jensen's.  Burkholder's  and  Gronwail's  inequalities 
we  can  see  that 


Ds.u 

and  sup 

K.u 

sup 

sguSl 


arc  in  LP(Q)  for  all  p<  x. 


X6 


R.  J.  ELLIOTT  AND  M.  KOHLMANN 


The  Poisson  random  measure  /i  is  a  special  case  of  an  integer  valued  random 
measure.  Therefore,  as  in  (3.18)  of  Jacod  [7],  there  is  a  set 

D  =  n  («J,  Ji)xZ)=ljcOx[0,7] 

and  for  each  Uo.t)eD  a  unique  point  /?,(ro)eZ  such  that  /i(a>,  {r}  xd:)  is  the  Dirac 
measure  at  /),(«.»).  Therefore, 

n(io,dt  x</r)=  V  <5(J>Wu))(r/r  x  dz). 

(w.s)D 

Consider  the  solution  flow  <i>((o,t,x0)  =  g 0,,(*o)  of  (1.1).  If  /r,  and  so  has  a 
jump  at  time  t.  the  magnitude  of  the  jump  of  <;  is  g(g0.i-(x o).z)>  where  c  =  /?,(w). 
Write  ,v  =  c0  ,-(x0).  Then  by  the  uniqueness  of  the  solution  of  (1.1)  we  have  the 
following  flow  and  semigroup  properties: 

s'o.  A*  o)  =  Zr.  Ax)  =  g,.  t(<;o..(*o)) 

=  Z'.Ax+g(x,:)lD). 
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Dq.t  —  0,.  r^o.i 
and 

Di.r  =  »0.  i'^oJ  •  (1-6) 


If  wc  consider  the  Eq.  (I.i)  for  soJ-^o)  and  the  equation  (1,3)  for  its  Jacobian 
00  l  as  a  single  system,  the  coefficients  are  not  bounded.  Following  Norris  [12]  we 
introduce  a  class  of  “lower  triangular"  coefficients.  See  also  Stroock,  [15], 

Definition  1.4  For  positive  integers  a,  d,  d, . dk,  with  d=d,  +  ■  ■  •  +  dk  write 

S,(</, . dk)  for  the  set  of  /V  eC{RJ  x  Z,  RJ)  of  the  form 


*(*.:)  = 


Xt2,(x\x2.:) 
\X'k)(x\x2 . x\:)J 


for  x  = 


:e  Rm.  where  RJ  is  identified  with  K‘,‘  x  •••  x  RJ\  s'eR*1  and  the  X  satisfy 


liA'll.v,».-,=suP  sup 

xeRJ  OS/>*<jS.V 


\iyj)iX(xL:)\ 

"(l+T?) 


V 


sup  \DjX%x,z)\e  L\Z)  rx  LX(Z) 

I 


for  all  positive  integers  N. 

Write  S(d . dk)  =  {J1S,(d< . dk). 

For  cocIFicients  not  involving  r  a  similar  space  S(d dk)  can  be  defined  as 
in  [6], 

Remarks  1.5  Note  (1.1)  and  (1.3)  can  be  considered  as  a  single  system  with  the 
coefficients  which  involve  z  belonging  to  S{d.d2)  and  the  coefficients  /l. 
(('vl/('4)0,which  are  independent  of  belonging  to  S(d,</2).  The  proof  of  Norris 
can  then  be  adapted  to  prove  the  following  result: 

Theorem  1.6  Suppose  g.  /i,  S2{ds . </*),  A  eS,(dl% _ dk)  and  inf,  del  |/  +gJ|>0. 

Then  there  is  a  map  <p:D  x  [0,  T]  x  [0,  T]  x  R^-'R'1  such  that 

i)  for  0 and  .vs Rd  <p(o>,s,t,x)  is  the  essentially  unique  solution  of  the 
equation 

dx,=gdg  +  hdv+  A  dt  (1.7) 


with  .v,  =  .v. 

ii)  for  each  o.sj  the  map  <p(to,sj ,  •)  is  Cx  in  .v  with  derivatives  of  all  orders 
satisfying  equations  derived  from  (1,7)  by  formal  differentiation, 
hi)  supw§„  fc'fsup.g^, |0"0(a>,4\M..v)|'’] 

£C{p.s.i.R.n.N.dl . dk,  a,  ||g|).V(1..v,.  |M|s<=,..v>.  |M||s„..v(). 


where  Dn  denotes  any  mixed  partial  operator  in  d/'xl....,(/cxll  of  order  less 
than  n. 
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Proof  This  is  a  technical  result  whose  proof,  using  smooth  truncations  of  the 
coefficients,  followinu  that  of  Norris  [12],  It  is  of  interest  to  note  that  condition 
(1.2) 

infdet  |/  + g,|>0 

X 

for  the  system  (1.7)  implies  the  corresponding  condition  for  the  enlarged  system: 
tlx,  =gdfi  +  hdv  +  Adt 

eg  5h  dA 

dDn  ,  =  ~  t>0.,  W  ^  D0',dv  +  -jg  D0  ,dt.  ( !  .8) 

The  function  corresponding  to  g,  that  is  the  coefficient  of  ji,  in  (1.8)  is 


and  the  state  space  now  consists  of  the  variables  x  and  D0 Recall  dg/dg  is 
another  notation  for  gt,  the  gradient  of  g,  so 


Forming  the  Jacobian  ofg  with  respect  to  x  and  D  we  obtain  an  operator  we  can 
formally  denote  by 


gxxD®D 


and,  working  in  coordinates,  we  see  det  |/  -t-^l  =  (det  |/  +g.xj3)-  Therefore, 
inf,  detj  /  +g,|  >0. 


2.  INTEGRATION  BY  PARTS 

In  this  section  we  shall  establish  an  integration  by  parts  formula. 

AsscstmoNS  2.1  Suppose  g,  h.  AeS1(dl,...,di), 

lim  0!g= 0  for  </^l,  (2.1) 

1:1-  < 

x„eRJ.  and  that  conditions  (1.2)  are  satisfied.  Consider  the  unique  solution  cj0i,(x0) 

lhi>  iviti'tli'Mi 

r 

so..(*o)  =  *o  +  J  j g(s:o.r  -  (-To).  :)fi{dr  x  dz) 

0  / 

+  \lHZo.,4XoUMdrxdz)  +  \  A(^r-(x0))dr  (2.2) 

o  /.  o 


for  0  <  /  <  I 
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Suppose  F:[0,  F]x  RJ-*R  is  a  bounded  C2  function.  Then  the  differentiation 
rule  gives 


'  C’F 

Zo.M  =  F( 0.  ,v0)  +  J  v- (r.  s:0_,j.V(,))  dr 

n  ( r 


'  CF 

+  i  J  (V=  <r’?o.r-(A0))i'(Co.,-(.Yo).e)d// 

o  y.  (  S 


1  vF 

+  J  J  It  ?o.r  •(v0))/i(c0,,  -  (.Vo). ;)  </'■ 

o  z  *  9 


•cF  . 

+  1  SS  (r-  Co.r  -(-Vo)M(Co.r  -  (-V0))  dr 


+  J7  ( ■ F(r-  Co. ,  -  (.<o)  +  Co, ,  •  (•'Co).  -*))  -  F(r ' ,  £o ,  -  (.v0) 

0  z 


CF 

-  (r  -So.r-iXoMCo.r  (Xol:)dfi 


-  F(0.  x0)  +  \  (~  +*1/;  a)  dr  +  if  ~  h  dv 

oVr  *C  J  hire 

+  J  J (/7(r-  Co.f  (x0)  +g(Co.,- (.v0), :))  —  F(r~, Ss0  ,  ,(.v0)) d/2 
0  / 

t 

+  J  J  ( a->  Co.r  (.v0)  + j?(c -(.V0), .-)) -F(r  ,?0r.  (.v0)) 

0  Z  ■ 

CF 

~  -<;o.r-ix0MZ0.r.{x0),:)dv.  (2.3) 


Consider  a  bounded  C1  function  c  ’.R^—'R  with  a  bounded  derivative  c,. 

The  random  variable  e(<fo.r(.Vo))  is  FT  measurable  and  for  0g/g7  we  can 
consider  the  martingale 


M,  —  £[f(Co.  Axo))  |  C  J. 


Writing  .v  —  so.i(  v0),  because  ti.e  process  c,  is  a  Markov,,  we  have 
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=  £[c(<;,.r(.v))] 

=  L'(l.v),  say. 

By  differentiating  with  respect  to  ,v  inside  the  expectation  we  see 


*.e 

ex 


—  £[^(<o.  t(xo))Do.t  I  F i ,]D0.l 

using  (1.6). 

Lemma  2.2  K(/..x)  is  differentiable  in  t. 

Proof  Consider  the  function  <f>(x)  =  c(il  T{x)).  Using  (2.3): 

0U\-j,.,(x))-</H-V)  =  J  A~dr  +  J  jh-—dv 

t-h  lX  t-h  i  OX 


(2.4) 


+  j  j(<t>(Zl-h.r-lx)+g(Zl-h'r.(x).:))-4>(Z,.Kr.(x)))d{i 

t-h  /. 


+  |  lm,-h.r  (x)+g(Z,->l'r-{x),:))-<P(Z,-H.r-yx)) 


-gtil  h.r  -  U). -)  (<?« MM*’- 


Taking  the  expected  value,  dividing  by  h  and  letting  /j  — *0  we  see 


eV  cV  eV 

-  ,-  =  A  ---  +  jh—d: 
ci  ex  b  ex 


‘'{f-.v  +  =,’(*, r)>  K(r. .X)  «{.x, r) ^ 

A  CV 

Because  the  martingale  K(/..v)  =  £[c(<;0  jCx0))|FJ  =  £[<  (<;,_  j(.x))]  is  differentiable 
in  f  and  .x  we  have  the  following  martingale  representation  result. 

Proposition  2.3  Write  y=^0_,(x0).  With  V(t,y)  =  £[<•(£,.  T(y))]  =  £[c(s'o.t(^o)) I F,] 
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K(f.Co.,(xo))=K(0,.vo) 

r 

+ 1 J ( Hr.  so.,  ( A-0)  +  g(Co.,  (a'o),  -))  -Hr'.  c0  ,  - (,v0)))  dft. 
o 


Proof  We  have  seen  above  that  V  is  differentiable  in  i  and  y.  Therefore,  writing 
down  the  differentiation  rule  (2.3)  for  V 


'  RV  1  3V 

Hi.  Co..(Xo»  =  ^(0,  -V0)  + 1  (  v  +  A  T?  )dr  + 1  J  h  77  dv 

0\or  cqj  0  j,  n, 

+  f  j ( Hr. c0., - (x0)  +  glfo.r -  (*o). :))- Hr’. Co., - Uo») d/1 
b  /. 

t 

+  MO/(c,Co.,-(x0)+«(Co.,-(x0).z)) 

0  7 

dV 

~  Hr'. Co.r-(xo)) -£(Co.,-(*o)> c)  Tv(r'.Co.,-(.x0)) dv 


i 


=  V'(0..vo)  +  J 

0 


dr 


t 


+  n  ( Hr.  Co.,-(*o)  +g(C o.,-(x0),  C))  -  »'(r",  Co.,-(  xo))  dfi. 
0  z 


(2.5) 


However,  from  Lemma  2.2  PVjcr  +  LV  =  0  and  so 


L(/,  y)  =  V{0.  ,v0)  +  {  J ( V(r.  C0., - Uo)  +  g(Co., - (x0), :))  -  Hr ' .  Co., •  (vo))  dfi- 
0  z 


The  integral  with  respect  to  the  compensated  measure  fi  is,  of  course,  a  martingale. 
Corollary  2.5  Nou\  in  particular,  the  representation 

T 

‘■(so.  lUo))  =  £[c(Co.  r(-xo))]  +  J  \  ( V(r,  C0.,-(x0)  +g(Co. ,-( xoK  =) 

oz 


~  Hr~.io.,-(x0))dfi. 
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Notation  2.6  Suppose  A(w.  t, :)  =  /.( t,:)  is  a  square  integrable,  predictable,  pos¬ 
sibly  vector-valued,  process  such  that  both  /i(f,  e)  and 


Hr.:) 


belong  to  L'(Z  x  [0,  T]  x  0) 
and  lim  A(f,r)=0,  for  almost  all  to. 

1:1- r. 


Consider  the  vector  valued  martingale 
L,  =  {J /(r,r)<//i 

QZ. 


(2.6) 


is 


0  7. 


For  \ 


<V. 


0/.(-k  0\^c-k 


(2-7) 


for  almost  all  w  by  (2.6). 

Therefore,  l,  =  j'0  j/il{r,:)d/i  for  almost  all  to. 

Tiicorem  2.7  Suppose  (j0-,(x0)  ls  rke  solution  of  Eq.  (2.2)  and  L,  is  defined  by  (2.7). 
Write  X  —  Co.r  ■(•*<>)•  Then 


£fr(Co.  ;(-vo))^/] 

=  -E 


where 


^(4o.r(-To))^o.r|  I />o.i-(/  +£*(•<•«))  ‘  lg^x.:)/.(r.z)n(d:,dr) 
0  / 


C’Z 


**“  U: 


f-l 


/V00/  With  F  given  as  in  Proposition  2.3  and  L  given  by  (2.7)  the  product  rule 
implies  that 

r  r 

yrLr  =  j  K-dL  +  f  Lr-  dV 


+  f  f(F(r,.v+g(x,  c))-  V(r 
0  / 


x))l(r.:)dp. 
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Taking  expectations  we  have 


/-(/(so.  ;(-vo))/-r]  — 1‘- 


V{r.x+g{x,  :))Hr,:)dv 


-E 


|  yjr.z)  d:\dr 


(2.8) 


and  the  final  expectation  is  zero  by  (2.6).  Furthermore,  because  !(r,:)- 

(fv./r'r, . 5/./dzm)  we  can  integrate  the  terms  by  parts  in  rt,...,;m,  respectively, 

and  obtain: 


f  V(r,x+g(x,:))l(r,z)dz'jdr 


=  -E 


T  dV 

Jj  —  (r.  x  +  g(x.  z))/.(r, z)  dz  dr 

0  7.°- 


—  —E 

~T  fiy 

f  j  —  (r,x+g{x.z))Mr,z)dfi 

=  -E 

"r  dV 

j  j  y  (r,y)g;(x,:)/.(r,:)dfi 

where 

_07  (~ 

y~io.,(xo). 

.or.  dy 

Here  we  have  again  used  (2.6),  and  dV/dz={dV/dz, . dV/d:m).  Recall 

F(r.  y)  =  £[c’(^0.  rC-^o))  I F,]  =  FMCi.tO’))]. 


and  from  (2.4) 


dV 

dy 


£[c<(^o.r(-vo))^o.r !  F 


so 


rr 

I 

o  \/ 


iff  V(r,x+g(x,z))l(r.z)dz^dr 


Writing 


=  -E 


1 f  E[c.(Z0'T(x0)D0,T\Fr']DS'g:(x,z)/.(r,z)dn 


0  7. 


and 


X ^-Lci(^oMxo))  ^o.  r  i  F r] 
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wc  have 


XrYr  =  }x,dYr  +  }  Y,.dXr 
0  0 


so 


EixTYT]  =  E 


j  XtdYt 


Substituting  in  (2.9), 


£[dso.  n-(oYl^rl 


-  -£  ci(€oMxo'i)Do.Tj  \  Do.lg.(x.:)/.(r,:)tt(d:,dr) 


0  z 


=  -£ 


t'A:o.  r(-vo))0  o.  rH  00' J  -  ( /  +&(*.  r))  ‘  'g.(x,  r)A(r,  :)ft(dz,  dr) 

0  7. 


Notation  2.8  Write 

wi./  =  j  J  Di.'  11  +g*)~lg,{X'=)g?(x.2)U  +gx)*‘lD?,;-ln(dz,dr) 

s  7. 

where,  as  above,  .x  =  <;0_,.(.v0)  and  *  denotes  the  transpose.  Then  M  is  the 
Malliavin  matrix  for  the  process. 

Corollary  2.9  In  Theorem  2.7  take  2(r,z)  to  be  the  process 

8tl  i+g,r ■'  Dt;,-1. 

Note  that  from  (2.1)  this  7.  satisfies  condition  (2.6).  Then  with  l(t.z)  =  (d/./dz)(t,z) 
and  Lr=IZ$/il(t.:)dfi 

=  ^[^(Co.  Axo))Do.tMo.t]-  (2.10) 

Remark  2 0  Ecjuoiion  (2.10)  cci m  be  considered  os  or.  i4integrotion'by'ponsM 
formula  for  c(<;()  r(x0)). 

In  Theorem  2.7  we  could  consider  an  /  which  is  just  the  derivative  of  7.  in  a 
single  direction,  say  /(/,z)  =  <v./?z,.  Then  with  L,  =  $,0jzHt,z)dfi  for  this  component 
the  above  calculations  go  through,  and  we  obtain  (2.9)  with  M0_T  replaced  by 

J'  |  DS.r.U+Kx) "  'gz,g*,(l+gx)*  ' Do.r- ll(dz,dr). 

O  / 
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Here  g.l=cg/d:l.  An  integration-by-parts  formula  (2.10)  is  obtained  which 
involves  the  derivative  of  g  in  just  a  single  direction.  However,  to  preserve 
symmetry,  we  prefer  formula  (2,10)  in  the  form  which  includes  the  gradient  g.. 


3.  HIGHER  DIMENSIONAL  FLOWS  AND  BOUNDS  FOR  DERIVATIVES 
Consider  a  product  function 

J  (Co.  rf  xo)) =  f(Co.  Ax  o))^(?o.  Axo)) 

satisfying  the  conditions  of  Theorem  2.7  and  apply  Corollary  2.9.  Then 


£  [/( Co,  A  A'o))  Lr] =  —  £  C(cs‘(Co.  Axo))k(£o.  Ax  o)  +  c(C  o.  Axo))kf(  Co.  Ax  o))^o,  o.  r]- 


What  we  would  like  to  do  in  (3.1)  is  take  k  =  Mq\Dq't  so  we  can  obtain  a  bound 
for  Cf.  However.  Z)0‘  lT  and  involve  the  past  of  the  processes  <;,  D  and  M.  This 
difficulty  can  be  overcome,  as  in  [6],  by  considering  augmented  flows  tp("\ 
n=  1,2,...,  defined  inductively  in  higher  dimensional  spaces.  Therefore  1.6  applies 
to  all  n S:  1,  so  that  analogs  of  Theorem  2.7  and  Corollary  2.9  may  be  used 
with  <p(t)  instead  of  <f>  and  k($m)  =  Mo.r^o.r  10  prove  the  following  result: 

Theorem  3.1  Suppose  e0Jx0)  is  the  solution  of  (2.1)  and  c  is  any  smooth  function 
with  hounded  derivatives.  If  M  of  is  in  some  Lf( Q) 

|£fr.(Co.jixo))]|^K  sup  |c(x)|. 

xc  Rd 


Remarks  3.2  As  is  well  known  (see  Malliavin  [10],  or  Stroock  [13])  the 
inequality  (3.4)  implies  the  random  variable  £0.AX o)  has  a  density, 

To  show  the  density  of  ^0.Axo)  >s  differentiable  we  must  obtain  bounds  of  the 
form 


E 


cfc'o.  Ax  o)) 


sup  |c(x)| 

xeR* 


where  ct  =  (<xl,<x2,...,ad)  is  a  multi-index  of  non-negative  integers  and 


c*c  _  d1'  d*1  (A- 


(3.1) 


An  argument  from  Fourier  analysis  (see  [12])  shows  that  if  (3.1)  is  true  for  all  a 
with  ja|=a,  +  ■■■  +a„£n  where  n'Zd+X  then  the  random  variable  c0.r(xo)  has  a 
density  d(x)  =  d(xt,...,xd)  which  is  in  C"~d~l(Rd). 
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If  wc  proceed  as  above  with  replacing  </>"’  and  consider  successively  higher 
derivatives  of  c  in  the  above  mentioned  analog  of  Corollary  2.9,  wc  can  derive  the 
following  result: 

Theorem  3.3  Suppose  the  inverse  of  the  Malliavin  matrix ,  is  1,1  every  LP(Q), 
1  ^  p  g  x .  Then  the  random  variable  io.tixo)  has  a  density  d{x)  which  belongs  to 
CX(RJ). 

Remarks  3.4  The  above  has  been  studied  in  detail  in  [6]  for  the  continuous 
case,  and  the  methods  carry  over  to  the  situation  here.  The  main  question, 
however,  concerns  the  existence  and  integrability  properties  of  Mq.t-  Now 
M16Lp(Q)  if  and  only  if  det  M~ 1  e  Lp(Cl),  and  for  the  symmetric  matrix  M  we 
have  the  following  inequality,  (see  [3]): 

(det  M) " ■ p  g  Hp)  -J  j  \x\Jt2p~  '  ’(exp  ( -  x*Mx)  dx. 

RJ 

To  show  the  integral  on  the  right  is  finite  delicate  Tauberian  theorems  are  applied 
in  [5]  and  [3].  The  conditions  for  these  to  be  satisfied  are  established  by  assuming 
g  satisfies  certain  growth  conditions.  The  objective  of  this  paper  is  to  give  the 
simple  proof  of  the  integration-by-parts  formula  (2.10),  which  does  not  involve  the 
calculus  of  variations  in  function  space,  and  indicate  how  this  simplifies  arguments 
concerning  the  existence  of  densities. 
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Abstract.  Using  stochastic  flows  and  the  Ito  differentiation  rule,  the  inte¬ 
grand  in  the  representation  of  a  martingale  as  a  stochastic  integral  is  identified. 
By  iterating  this  representation  result  a  homogeneous  chaos  type  expansion 
is  obtained.  Using  the  stochastic  integral  representation,  an  integration  by 
parts  formula  is  obtained  without  using  any  calculus  of  variations  in  function 
space.  If  the  inverse  of  the  Malliavin  matrix  belongs  to  all  spaces  //(ft)  it 
follows  that  a  random  variable  has  a  smooth  density. 


1.  Introduction 

The  Malliavin  calculus  was  originally  developed  in  a  remarkable  paper,  [7],  as 
a  calculus  of  variations  in  function  space;  one  of  its  applications  is  to  show  that 
under  appropriate  conditions  diffusion  processes  have  densities.  Bismut,  in  [3], 
approached  the  Malliavin  calculus  by  using  stochastic  flows  to  describe  perturba¬ 
tions  of  trajectories  while  Stroock  in,  for  example,  [9]  used  more  functional 
analytic  methods.  Simplifications  of  the  Malliavin  theory  were  provided  in  the 
papers  by  Bichteler  and  Fonken  [2]  and  Norris  [8],  and  in  the  recent  book  by 
Bell  [1].  A  very  readable  exposition  can  be  found  in  the  paper  by  Zakai  [10], 
and  a  careful  treatment  is  in  the  text  by  Ikeda  and  Watanabe  [5].  In  all  these 
presentations  function  space  calculus  is  used.  A  contribution  of  this  paper  is  that, 
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for  Markov  diffusions,  some  of  the  initial  results  of  the  Malliavin  theory,  including 
the  “integration  by  parts"  formula,  are  obtained  by  techniques  which  involve 
differentiation  with  respect  to  initial  conditions. 

The  paper  begins  by  using  the  theory  of  stochastic  flows  to  identify  the 
integrand  in  a  stochastic  integral.  After  some  rearrangement  this  integrand  is  P 

itself  written  in  terms  of  a  martingale  which  can  be  expressed  as  a  stochastic 
integral,  and  by  recursively  repeating  the  representation  a  homogeneous  chaos 
expansion  is  obtained.  Using  the  stochastic  integral  representation,  an  integration 
by  parts  formula  is  then  derived.  If  the  inverse  of  the  Malliavin  matrix  M  belongs 
to  all  the  spaces  Lp{Cl)  we  show  that  a  random  variable  has  a  smooth  density;  ^ 

however,  the  difficult  questions  concerning  the  relationship  between  Hormander’s 
conditions  on  the  coefficient  vector  fields  and  the  integrability  of  M-1  are  not 
discussed  in  this  paper.  This  paper  was  presented  at  the  Workshop  on  Diffusion 
Approximations  held  at  the  International  Institute  for  Applied  Systems  Analysis,  p 
Laxenburg,  Austria,  in  July  1987.  A  fuller  treatment  of  the  ideas  given  here  can 
be  found  in  [4], 


2.  Dynamics 


Consider  a  stochastic  differential  system 

dx,  =  X0{t,  x,)  dt  +  Xj(t,  x,)  dw[.  (2.1) 

Here  xeRd,  Os/sT,  and  w  =  (m'1,  . . . ,  wm)  is  an  m-dimensional  Brownian 
motion  on  {Cl,  F,  P).  We  shall  assume  that  the  coefficient  vector  fields  X  are 
smooth  and  have  bounded  derivatives  of  all  orders. 

From  results  in  [5],  for  example,  it  is  known  that  forO rs  T and  xse  R‘‘ 
there  is  a  unique  solution  &,,(x ,)  of  (2.1)  with  £,,(*,)  =  xs.  Furthermore,  there  is 
a  version  of  this  solution  which,  almost  surely,  is  smooth  in  xs  e  R‘‘. 

If  x0e  R'1  and  x-^0,{x0),  because  the  solutions  of  (2.1)  are  unique, 

£o,t(*o)  =  £<,r(£o.i(*o))  =  £i.t(x)‘  (2.2) 


Write  Dsl=dt;si/dx  for  the  Jacobian  of  the  map  x->  fjs,(x).  Then,  differentiating 

(2.2), 

Dot  —  A.rAj.i  • 

Again,  from  [5]  we  know  that  D  satisfies  the  equation 


dDv-~*Dv  dt  +  ^j  D,  ,  dw\ 


(2.3) 


with  Dss  =  I,  the  d  x  d  identity  matrix. 

Consider  the  matrix  function  VSJ  defined  by  the  stochastic  differential 
equation 
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with  VStS  =  I.  Here 


Then,  see  [2],  d{VslDSJ)  =  0  so 

vsj  =  d;J. 


3.  Martingale  Representation 


Suppose  x0 eR‘‘  is  given.  Consider  a  smooth,  bounded  function  c  on  R'1  and 
the  random  variable  c(g0.r(x o))-  Write  {F,}  for  the  right  continuous,  complete 
filtration  generated  by  F,  =  /}.  Because  x0  is  known  tr{.y, 

and  the  process  (x„  tv,)  is  Markov.  Consider  the  martingale 

M,  =  F[c(f0.r(-v0))|F,]. 


Then  by  the  martingale  representation  result 

M,  =  M0+\  y,(s)  dw‘s  (3.1) 

Jo 

v 

for  some  predictable,  square  integrable  process  y.  However,  because  go.,(xo)  is 
Markov,  writing  *  =  f0.i(xo), 

M,  =  E[c(6,t(x0))M  =  BWfoW)]  =  F[c(<f,r(x))|FJ  =  V(t,  x). 


By  the  chain  rule  c(|,  T(x))  is  differentiable  in  x.  Consequently,  V(t,  x)  is 
differentiable  in  x.  By  considering  the  backward  equation  for  £,iT(x)  as  in  [6]  we 
see  V(t,x)  is  differentiable  in  t.  Therefore,  applying  the  Ito  differentiation  rule 
to  V(t,  x)  with  x  =  &,»(*<>), 


nUo.,(*o))=  V(0,.vo)  + 


0\ds  )  Jo  dx 


Here 


''  ■  d  1  d  (  m  \  d2 

L=lx'^2  1  z 

1-1  oXi  f  oXidXj 


(3.2) 


However,  V(t,  g0,(xo))  =  M,  so  the  decompositions  (3.1)  and  (3.2)  must  be  the 
same.  The  bounded  variation  term  in  (3.2)  is,  therefore,  zero,  i.e., 


dV 

ds 


+  LV  =  0 


and  (as  is  well  known)  V  is  the  solution  of  the  backward  Kolmogorov  equation 
with  a  final  condition 

c(xT)  =  V(T,  xT). 
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Equating  the  martingale  terms  in  (3.1)  and  (3.2) 


Differentiating  inside  the  expectation 
dV 

—  =  £[cf(6.r(.v))D,.T|£,]  (by  the  chain  rule) 

=  F  [  ^  ( fo.  r(  -vo) )  D0.  t!  F  ]  D0.! . 

So 


7/(0  —  E  [  ( fo.  r  ( -vo) )  D0.  r  i  £ »  ]  D0, !  Xt 


(3.3) 


and 


M,  =  £[c(&.T(*b))]  + 


D[tV  (4r(.Vo))Do.r|FJ)D0-jX,(i>  &Jx b))  dw\ .  (3.4) 


Remark  3.1.  Note  the  term  £[C{(&,r(*o))D0>T|Fs]  is  itself  a  martingale.  If  the 
representation  is  written  down  at  t  =  T 


Mr  —  c(&).r(xo)) 


-  D(c(£0.t(wo))]  + 


£tQ(fo.T(.Vo))D0,r|FJD0-j^  dw{ . 


(3.5) 


Also,  the  representation  (3.4)  holds  for  vector  (or  matrix)  functions  c. 


If  we  take  c(£)  =  £  to  be  the  identity  map  on  R‘',  (3.5)  gives 

£o.r(*o) =  F[£o,r(*o)3+  f  F  [  D0,  rj  FS]D()\X ,  dw's . 

Jo 

Also,  if  we  consider  (3.5)  for  a  second  smooth  bounded  function  g  and  take  the 
expected  value  of  the  product  of  each  side,  we  see 

£[c(£o.r(a'o))lK£o.T(-vo))] 

=  £[C(fo.T(x0))]£[g(&.r(Jr0))] 

+  £  [  I  f 7  E[cp0,T\FADZX,XfDi;'E[g(D0,r\F,f  ds].  (3.6) 
L i  =» i  Jo 


Definition  3.2.  The  Malliavin  matrix  for  the  system  (2.1)  is 

du) . 

/ 

Note  something  resembling  M0, ,  occurs  in  (3.6). 


M$,t=  £  (f  DZlX1(u)Xf(u)Dt:' 
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4.  Homogeneous  Chaos  Expansions 

Consider  an  enlarged  system  with  components  £(l)  =  (£,  D).  The  stochastic 
differential  equation  for  £(I)  is,  therefore,  the  system  (2.1)  and  (2.3).  The 
coefficie:  ts  in  (2.3)  are  no  longer  bounded,  but  following  Norris  [8]  a  sequence 
of  “triangular”  systems  can  be  considered  and  the  results  on  stochastic  flows  still 
hold.  We  can,  therefore,  consider  the  Jacobian  D(1)  of  the  system  f(,)  and  a 
system  |(2)  =  (£0),  £>(l)).  Proceeding  in  this  way  £(n)  is  a  system  with  components 
(£(n",),  D("",)j.  Write 

cu)_i£  D 

C  ~ d£DoT ' 

(i) 


(4.1) 


m 


§§§ 


.-«>  dc  n(i)  tr 

C  ~  gft't  U0.T,  etC- 

Equation  (3.4)  can  then  be  written 

cfe.r(.vo))  =  £[c(#o.r(.Vo))]+  f  E^F^D  £X,  dw[. 

Jo 

However,  £[c<i>(^o|t-)|Fj]  can  be  represented,  as  in  Section  3,  as  a  stochastic 
integral 

£[c',,!FJ  =  £[c(1,]+  P  £[c<2>|FJl]£)o|,,j',X),>(S|)  dw{t. 

Jo 

Here,  Xjn  is  the  coefficient  vector  field  of  wj  in  the  system  defining  £(,). 
Substituting  in  (4.1) 

c(&r(.v0))  =  E[c]  +  £[c<n]  [  Do\Xt  dw‘s 

Jo 

+  (£  Elc^FJD^'xyW  Do,\Xt dw‘s.  (4.2) 

Now  E[c{2)\Fh]  can  be  expressed  as  a  stochastic  integral  and  the  result  substituted 
in  (4.2).  Proceeding  in  this  way  we  obtain  the  homogeneous  chaos  expansion  of 
the  random  variable  c(5>jT(x0)).  The  repeated  stochastic  integrals  do  not  involve 
c  but  only  the  Jacobians  D(k)  and  coefficients  Xw. 

5.  Integration  by  Parts 

Lemma  5.1.  Suppose  «  =  («,,...,  i/m)  is  a  square  integrable  predictable  process. 
Then 

rT  i  »«  r  rT  i 

£1  c(&.r(* o))  J  Hi dw\ J  =  £  £  |^(f0.T(*o))A).T  Do',X,(s)u;{s)  ds  j  . 


-c7.  - 


V 


t'  .•f'.t'i  | ,  i,"* ,  l  \\'  il  rtltfliV  V|>’  i", 


no 
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Proof.  Consider  the  representation  (3.5)  for  c(£0,t(*o))-  Multiply  by  u,dw's 
and,  using  Fubini’s  theorem,  take  the  expectation.  □ 

Corollary  5.2.  Take  u,(s)  =  (Do,iX(s))*.  Then 

£[c(&t(.v0))  j  (■ Do.lX,(s ))*  dw\  j  =  E[c((^r(x0))DoxMox].  (5.1) 


Remark  5.3.  Consider  a  product  function  h(£0T(xo))  =  c(£0,T(*o))g(£o.T(*o))  and 
apply  Corollary  5.2  to  h.  Then 

£^(<£)(&,t(*o)) 

We  would  like  to  take  g  =  Mo.tDo.t  in  (5.2)  so  that  we  can  obtain  a  bound  for 
c(.  This  can  be  done  by  considering,  again  following  Norris  [8],  a  hierarchy  of 
stochastic  systems  similar  to,  but  different  from,  those  introduced  in  Section  4. 

This  time  write  <p<0,(h’,  s,  t,  *)  =  £,_,(*)  for  the  flow  defined  by  (2.1)  and 
D{£}(x)  =  D,,(x)  for  its  Jacobian.  R{°}= J'(D",'X; (•<))*  dw'u  and  Mf]  ~  MSJ  is 
the  Malliavin  matrix  defined  in  (3.2).  Note  that  M,-(  can  be  considered  as  the 
predictable  quadratic  variation  of  the  tensor  product  of  R(0)  with  its  adjoint,  that 
is  M‘°I,  =  <R<0,®/?<0,V 

Now  consider  an  enlarged  system  <p(,)  with  components 
=  DW,R<°\MW). 

The  results  of  Norris  [8]  on  stochastic  flows  allow  us  to  discuss  the  Jacobian 
D(l>  of  <pU).  Assume  X-11  is  the  coefficient  of  w'  in  the  system  describing 
and  write 

/?1V  =  |  (D(s'Jr'X\'\u))*  dw‘u, 

M<”  =  <Rn)®R,0>V 

Then  define 

and  inductively,  =  (p<n),  D{n),  R(n\  Write  Vn  for  the  gradient 

operator  in  the  components  of  (p(n) .  The  following  result  is  established  like 
equation  (5.2)  by  considering  the  martingale  representation  (3.5)  of  the  produce 
eg- 


J  (Do.[Xt)*  dw[  j  =  £[(c?g  +  cgf)D0.TM0.r]. 


Martingale  Representation  and  the  Malliavin  Calculus 


111 


Corollary  5.5.  Gronwall's  inequality  shows  that  D  1  is  in  all  the  Lp(ft)  spaces , 
so  if  Mq'T  is  in  some  Lp(fl)  taking  g(<pm( 0,  T))  =  Moj-E>o.t  in  (5.3) 

£[cf  (fo.r(-vo))]  =  E[c(€o.t(xo))  M  o'tDo't®  Ro,t] 

-E[c(^r{x0))^l8)(D0_r,M0,r)DixirM[%]. 

Because  c  is  bounded  we,  therefore,  have  the  following  result: 

Theorem  5.6.  Suppose  £o.r(*o)  Is  the  solution  of  { 2.1)  and  c  is  any  smooth  bounded 
function  with  bounded  derivatives.  Then  ifMo'r  is  in  some  Lp(ft) 

|£0«(&.t(*o))]|  -  K  SUR  |c(jf)|.  (5.4) 

xe.Rd 

Remark  5.7.  It  is  well  known  that  (5.4)  implies  that  the  random  variable  £>.r(*o) 
has  a  density  d(x).  To  show  the  density  d  is  smooth  we  wish  to  establish 
inequalities  of  the  form 

£||c£(&.t(xo))  K  sup  |c(x)|.  (5.5) 

1  L9?  J  xeRJ 


dn  d«i  aaj 

dF~dtf'W: 

An  argument  from  Fourier  analysis  (see  [8])  shows  that  if  (5.5)  is  true  for  all  a 
with  |«|  =  a,  +  -  •  ■  +  all^n  where  n>d  + 1,  then  the  random  variable  f0.r(*o) 
has  a  density  d(x)  which  is  in  C""‘,“l(i?‘,). 

Apply  Corollary  5.5  to  c(  rather  than  c  so 

£[c«(4rdo))]  =  ^[^(^o.rf-Vo))^  o.r-Oo.r  ®  ^o.r] 

~  £[c{(&.t(^o))(^i8)( Dq,t>  Wo.r)£^o!rMo!r]-  (5-6) 

Consider  the  two  terms  on  the  right  of  (5.6)  and  write  M  =  M0%T,  D  =  D0  T,  etc. 
Let 

gI(<P<1,)  =  M'lD_l®^/Vr,C"1 

and 

g2(cpi2))  =  WigHD,  M)Da)Mu>M~,D~'. 

Applying  Theorem  5.4  to  eg,  and  cg2, 

£[c(&.t(.v0)W 

+  E[C(t0'T(Xo))(V2glK<p™)DmM(2))  (5.7) 

and 

E\c((n rW)?:!?(!,)®J?]  =  E[c((f0'T(x0))(T,g)(D,  M)D<:}M<;; J 

+  £[c($,.TUo))(V3g2)(9U))£?(3>M(3)].  (5.8) 


:  ■’SKs 5  i  ’  S  y'if:  a  f 
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Using  (5.7)  and  (5.8)  the  terms  on  the  right  of  (5.6)  can  be  replaced  by  terms 
involving  c.  This  procedure  can  be  iterated  using  Theorem  5.4  and  the  following 
result  established: 

Theorem  5.8.  Suppose  M~'  is  in  all  spaces  Lp(Cl),  l<p<oo.  Then  the  random 
variable  £0.t(-yo)  has  a  smooth  density. 

The  remaining  questions  concern  the  existence  and  integrability  properties 
of  JWo.t-  These  have  been  carefully  studied  (see  [5]  or  [8],  for  example).  In  fact 
Mo.r  is  in  Lr(ft)  for  all  p,  l<p<oo,  if  the  following  condition  of  Hormander 
is  satisfied: 

Condition  5.9.  The  vector  space  V(.v0)  generated  by  the  coefficient  vector  fields 
X, .... ,  Xm  and  the  brackets  [X„  Xj),  0  £  ij  £  m,  [X„  [ Xj ,  XJ],  0  <  ij,  k  <  m, 
etc.,  evaluated  at  .v0e  Rd,  is  the  whole  of  R'1. 
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Summary 


The  adjoint  process  and  minimum  principle  for  a  partially  observed  diffusion 
can  be  obtained  by  differentiating  the  statement  that  a  control  u*  is  optimal.  Using 
stochastic  flows  the  variation  in  the  cost  resulting  from  a  change  in  an  optimal  control 
can  be  computed  explicitly.  The  technical  difficulty  is  to  justify  the  differentiation. 


1.  INTRODUCTION. 

Using  stochastic  flows  we  calculate  below  the  change  in  the  cost  due  to  a  ‘strong’  variation 
of  an  optimal  control.  Differentiating  this  quantity  enables  us  to  identify  the  adjoint,  or  co¬ 
state  variable,  and  give  a  partially  observed  minimum  principle.  If  the  drift  coefficient  is 
differentiable  in  the  control  variable  the  related  result  of  Bensoussan  [2]  follows  from  our 
theorem.  Full  details  will  appear  in  [1].  The  method  appears  simpler  than  that  employed  in 
Haussman  [4]. 
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2.  DYNAMICAL  EQUATIONS. 

Suppose  the  state  of  a  stochastic  system  is  described  by  the  equation 

d(t  =  f(t,£t,u)dt  +  g(t,tt)dwt, 

£teRd,  fo  =  zo,  0  <t<T.  (2.1) 

The  control  variable  u  will  take  values  in  a  compact  subset  U  of  some  Euclidean  space  Rk. 

We  shall  assume 

A\:  x0  e  Rd  is  given. 

A?:  f  :  [0,  T]  x  Rd  x  U  — *  Rd  is  Borel  measurable,  continuous  in  u  for  each  (t,  x ),  continuously 
differentiable  in  x  for  each  (t ,  u)  and 

(1  +  |z!)-1|.f(i,i,u)|  +  \f,(t,x,u)\  <  Ki. 

A,:  g  :  [0,  T]  xRd  ->Rd®  Rn  is  a  matrix  valued  function,  Borel  measurable,  continuously 
differentiable  in  x,  and  for  some  Ki'. 

The  observation  process  is  defined  by 

dyt  =  h(Zt)dt  +  dut  (2.2) 

yt  6  Rm,  y0  =  0,  0  <  t  <  T. 

In  (2.1)  and  (2.2)  w  =  (w1 . .  ,wn)  and  u  =  (i/1,...,*/®1)  are  independent  Brownian 
notions  defined  on  a  probability  space  (fl,  j P,  P). 

Furthermore,  we  assume 

A±:  h:  Rd  Rm  is  Borel  measurable,  continuously  differentiable  in  x  and 

|/i(f,x)|  +  \hx(t,x)\  <  Kz. 

REMARK  2.1.  These  hypotheses  can  be  weakened  to  those  discussed  by  Haussman  [4]. 
See  [1]. 

Write  P  for  the  Wiener  measure  on  C([0,T],Rn)  and  fi  for  the  Wiener  measure  on 
C([0,T),Rm). 


n  =  C([0,T],Rn)xC([0,T],Rm) 


and  the  coordinate  functions  in  SI  will  be  denoted  (xt,yt).  Wiener  measure  P  on  is 


P(dx,dy )  =  P(dx)jj,(dy). 


DEFINITION  2.2.  Y  =  {Ft}  will  be  the  right  continuous,  complete  filtration  on 
C([G,T},Rm)  generated  by 

yt°  =  °{y»  :  5  <  t}. 

The  set  of  admissible  control  functions  JJ_  will  be  the  Y -predictable  fimctions  defined  on 
(0 ,T)  x  C([0,T],Pm)  with  values  in  U. 

For  u  £  U  and  x  £  Rd,  £“ t(x )  will  denote  the  strong  solution  of  (2.1)  corresponding  to 
u  with  =  x. 

Define 

ZU*)  =  «P  (J‘ C,W)'4  -  \  j\aAx)fiv) .  (2.3) 

Note  a  version  of  Z  defined  for  every  trajectory  y  can  be  obtained  by  integrating  the  stochastic 
integral  in  the  exponential  by  parts. 

If  a  new  probability  measure  Pu  defined  on  Q  by  putting 

dPu 

under  Pu  (£o,t(xo),  Vt )  is  a  solution  of  the  system  (2.1)  and  (2.2).  That  is,  under  Pu,  £o,t(xo) 
remains  a  strong  solution  of  (2.1)  and  there  is  an  independent  Brownian  motion  v  such  that 
yt  satisfies  (2.2). 

Because  of  hypothesis  A 4,  for  0  <  t  <  T  easy  applications  of  Burkholder’s  and  Gronwall’s 
inequalities  show  that 

miiMYi  <  00  (2.4) 

for  all  u  6  U_  and  all  p,  1  <  p  <  00. 

COST  2.3.  We  shall  suppose  the  cost  is  purely  terminal  and  equals 


c(£o,t(xo)) 


nrh 


orA  s*  10 


m  k/VUUU^U 


d,  differentiable  function.  If  control  u  6  U  is  used  the  expected  cost  is 


J(u)  = 


With  respect  to  P,  under  which  yt  is  a  Brownian  motion 


J{u)  =  .E(Z0“T(*oM«ou,r(zo))]- 


(2.5) 


A  control  u*  6  ]J_  is  optimal  if 


J(u*)  <  J(u) 

for  all  u  £  U_.  We  shall  suppose  there  is  an  optimal  control  u*. 


3.  FLOWS. 


For  u  6  U_  and  x  €  Rd  consider  the  strong  solution 

£J,l(*)  =  X  +  j'  f(r<CAX)’V')dr  +  j'  S(r'  (tAX))dmr' 


(3.1) 


We  wish  to  consider  the  behaviour  of  ^it(x)  for  each  trajectory  y  of  the  observation  process. 
In  fact  the  results  of  Bismut  [3]  and  Kunita  [6]  extend  and  show  the  map 


C.,  -Rd  Rd 


is,  almost  surely,  a  diffeomorphism  for  each  y  €  C([0,T],i?m). 

Write 

||C(2o)||t=  sup  |ejf#(*0)|. 

0<.<l 

Then,  using  Gronwall’s  and  Jensen’s  inequalities,  for  any  p,  l  <  p  <  oo 

T 

I!£U(xo)||t  <  ^(l  +  |Zo|P  +  |  J  ff(r , f  0,r(x0 ))dwr jr j 

almost  surely,  for  some  constant  C. 

Using  A3  and  Burkholder’s  inequality 


H£w(xo)||t  €  Lp  for  1  <  p  <  00. 


Suppose  u*  is  an  optimal  control,  and  write 


£«(•)  for 


The  Jacobian  Hz*-  is  the  matrix  solution  Ct  of  the  equation 


dc,  =  /,(<, u')c,dt + 


i= 1 


with  Ct  =  I. 


(3.2) 


'4 ,  1  iK 


Here  g ^  is  the  zth  column  of  g  and  I  is  the  n  x  n  identity  matrix.  Writing  ||C||r  = 
suPo<j<(  |C*|  &nd  using  Burkholder’s,  Jensen’s  and  Gronwall’s  inequalities  we  see  |jC||r  G  Ip, 
1  <  p  <  oo. 

Consider  the  matrix  valued  process  D  defined  by 


D,  =  I-  f  DrMr,(Zr{x),  K)dr 

-  £  f  Dr4i>(r,CJ*))i»,r  +  E  fDr(9iH r.{*  r(*»)!dr  (3.3) 

«=1  ^  t=l  -71 


Then  as  in  [5]  or  [6]  d(DtCt )  =  0  and  DjCj  =  I  so 


Furthermore,  ||D||*  €  Fp,  1  <  p  <  oo. 

Suppose  2t  =  J*Hidw'r  is  a  d-dimensional  semimartingale.  Bismut  [3] 

shows  one  can  consider  the  process  £• >t(z( )  and  in  fact: 


C,l(2«)  =  Z‘  +  jt  (/('''  £,r(Zr),<) 

+E#(r.«,r(^)iu:)  %*.• 


+  5  £#(*.*■))* 


t=2 


(3.4) 


DEFINITION  3.1.  For  s  6  [0,T],  A  >  0  such  that  0  <  s  <  $  +  h  <  T,  for  any  u  E  U,  and 
A  €  Yt  consider  a  ‘strong’  variation  u  of  u*  defined  by 


ii( i.  u>) 

\  j  ^  ; 


j  u*(t,w)  if(t,w)£[$,s  +  h]xA 
\  u(t,w)  if  (t,w)  €  [s,s  +  h]  x  A. 


THEOREM  3.2.  For  any  strong  variation  u  of  u *  consider  the  process 


Z‘=X  +  I,  ■^r’  G.r(*r)»  «r)  -  /(r.  C,r(Zr),  <))*. 


(3.5) 


Then  the  process  £*it(zt)  is  indistinguishable  from  ^jt(x). 

PROOF.  We  shall  substitute  in  (3.4),  (noting  H{  =  0  for  all  i).  Therefore, 

C, <(*.)  =  *  +  j (V(r,  £!,(«,.),<)* 

+ jf  <•;))* 

+  J  S(r,C,,(*r))*Or. 

The  solution  of  (3.1)  is  unique,  so  t(z<)  =  £“,(: r).  Note  u(t)  =  u*(t)  if  t  >  s+h  so  zt  -  zs+h 
if  t  >  s  +  h  and 

=  Cf*,i(C.H.*(*))-  (3-6) 


4.  THE  EXPONENTIAL  DENSITY. 


Consider  the  (d  +  l)-dimensional  system 


C,t(x)  =  *  +  jf  /(»-,G,r(*).«?)*  +  J  g{r,Ct>r(x))dwr 

Zl,t(X* Z)  =  Z  +  J  Zt,r(x*  z)h(C,r(*))'dyr'  (4.1) 

That  is,  we  are  considering  an  augmented  flow  (f ,  Z)  in  in  which  Z*  has  a  variable 


initial  condition  z  €  R.  Note: 


%*,t{x iz)  ~  z^t,i(x)‘ 


The  map  (x,  z)  — » (C,<(2),  Z»,t(x> z))  is,  almost  surely,  a  diffeomorphism  of  Rd+1.  Clearly, 

9 C,t  n  df  dg 

oz  oz  oz 

The  Jacobian  of  this  augmented  map  is  represented  by  the  matrix 


az  r 

dx 


az: 

dz 


In  particular,  from  (4.1),  for  1  <  :  <  d 


dz:,t 

dxt 


T'  .  ^iV  I  ui  ( c*  /_\\ 


We  are  interested  in  solutions  of  (4.1)  and  (4.2)  only  when  2  =  1,  so  as  above  we  write 


Z;it(x)  for  Z*t(x,  1)  etc. 


Lemma  4.1. 


gg£«  „ 

dx  z *' 


=  Zlt(X)(ja  MC,i(*))  •  -jf-dVr) 


where,  as  in  (2.2),  dvt  =  dyt  -  h(£*  t(x))dt. 
PROOF.  From  (4.2) 


dz;,t 

dx 


*  rt 


Write 


Then 


and  the  product  rule  gives 


/r  qc* 

(  d  (^i, r(x))  +  Z»,r (x)dx(^s,r(x))~~^~ Jdyr 

z.*,,W  = 1  +  frvW(SA*Wv r 

gives 

L*AX)  =  £  L*Ax)h'(Ct,r(x))dyr 

+  J  z:.r (*)>>,  •  %%- 


Consequently,  LSit(x)  is  also  a  solution  of  (4.3),  so  by  uniqueness 


L*Ax)  = 


dz;,t 


LEMMA  4.2.  If  zt  is  as  defined  in  (3,5) 


z:m) = 


PROOF. 


2,»  =  i  +  f'KAx)h’M.r(*))<iyr- 


Applying  (3.4)  to  Z*>t(zt)  we  see: 

z;.<(*r)  =  1  +  f  Z:A*r)h'((:A*r))dyr 

=  1  +  f  ZlA^rWCA^r 

by  Theorem  3.2.  However,  (4.4)  has  a  unique  solution  so 


Again,  note  that  for  t  >  s  +  h 


=  z.». 


z;.,(*,) = z;, ,(*.+»)• 


(4.5) 


5.  THE  ADJOINT  PROCESS. 


u*  will  be  <*n  optimal  control  and  u  a  perturbation  of  u *  as  in  Definition  3.1.  Again  write 


x  —  fo.jfao)- 


The  minimum  cost  is 


J(u*)  -  E{ZliT{xQ)c{QtT{xQ))) 

=  E[Z^xo)Z:tT(x)c(Cs>T(x))). 


Also, 


J(u)  =  E[Zlt(x0)ZZT(x)c(a,T(z))) 

~  E[Zo,t(xo)Z»,T(z*+h)c(£,T(z*+h))} 

by  (3.6)  and  (4.5).  Recall  Z*T(-)  and  c(f*y(-))  are  d.fferentiable  almost  surely,  with  contin¬ 
uous  and  uniformly  integrable  derivatives.  Consequently,  writing 

r(s,*r)  =  Zo*,.(^o)z;,r(zr){c<(f;,T(zr))  (»r) 


for  s  <  r  <  s  +  h,  we  have 


J(u)  -  j(u-)  =  s[z;..(*0){z;, ,(».+*><«, .(*«■»))  -  2,»*»)}1 

=  js[  /  ns,zr)(f(v,C,r(zr),ur)  -  /(r, <))*].  (5.1) 

This  formula  describes  the  change  in  the  expected  cost  arising  from  the  perturbation  u  of 
the  optimal  control.  However,  J{u)  >  J(u*)  for  all  u  6  U.  so  the  right  hand  side  of  (5.1)  is 
non-negative  for  all  h  >  0.  We  wish  to  divide  by  h  >  0  and  let  h  -*  0.  This  requires  some 
careful  arguments  using  the  uniform  boundedness  of  the  random  variables  and  the  monotone 
class  theorem.  It  can  be  shown  that  there  is  a  set  S  C  [0,  T]  of  zero  Lebesgue  measure  such 
that  if  s  $  S 

E[T(s,x)(f(s,Q,^o),u)-f(s,Co,.^o),u:))lA}>0  (5.2) 


for  any  u  €.U  and  A  €  Ya. 

Details  of  this  argument  can  be  found  in  [1].  Define 

p,(x)  =  E-  [c{«;,r(*o)) 

+  d.(o,T(X° ))(jf  A(Ko,«(Zo))  r,v{l)] 

where  x  —  £ o,s(xo )  and  E *  is  the  expectation  under  P*  =  Pu*. 

In  (5.2)  we  have  established  the  following: 


THEOREM  5.1.  p3(x )  is  the  adjoint  process  for  the  partially  observed  optimal  control  prob¬ 
lem.  That  is,  if  u*  €  H  is  optimal  there  is  a  set  S  C  [0,  T]  of  zero  Lebesgue  measure  such 
that  for  s  £  S 

E*{pt(x)f(s, X, tf *)  I  7,]  >  E*\p,(x)f($, X, u)  I  Yt]  a.s.  (5.3) 


so  the  optimal  control  u*  almost  surely  minimizes  the  conditional  Hamiltonian. 

If  x  =  titt(x0)  has  a  conditional  density  qt(x)  under  P*,  and  if  /  is  differentiable  in  u , 

/f"  aN  •  i* 

(i ).o)  linpues 

Y^(ui(s)  -  u'i(s))  T(s,  x)  ~  (s,  x, u*)q,(x)dx  >  0. 

This  is  the  result  of  Bensoussan  [2]. 
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ORDINARY  DIFFERENTIAL  EQUATIONS  AND  FLOWS 


Robert  J.  Elliott 
University  of  Alberta 

Department  of  Statistics  and  Applied  Probability 
and  Applied  Mathematics  Institute 


1.  INTRODUCTION. 

The  theory  of  stochastic  flows  was  first  developed  in  the  works  of  Kunita  [5]  and 
Bismut  [2];  they  have  been  used  to  discuss,  for  example,  stochastic  control  [l]  and  the 
Malliavin  calculus  [4].  However,  some  of  the  corresponding  ideas  concerning  deterministic 
flows  do  not  appear  so  well  known  to  those  working  in  ordinary  differential  equations, 
although  they  are  probably  familiar  in  terms  of  vector  fields  and  their  pull-backs  to  differ¬ 
ential  topologists. 

2.  DYNAMICS. 

Consider  an  ordinary  differential  equation 

d(t=f(t,()dt  (2.1) 


where  £  6  Rd  and  t  >  0. 

For  simplicity  we  shall  suppose  /  :  [0,  oo)  x  Rd  — »  Rd  is  C°°  and  of  linear  growth. 
Write  (x)  =  £8)t  (x)  for  the  unique  solution  of  (2.1)  which  is  such  that  £8)8  (x)  =  x,  i.e., 

t»JL  (x)  =  x  +  J  /(u>  6,«  [x))du.  (2.2) 

Partially  supported  by  NSERC  grant  A-7964  and  the  Air  Force  Office  of  Scientific  Research, 
United  States  Air  Force,  under  grant  AFOSR-86-0332. 
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Then  it  is  known  the  map 


is  differentiable.  Write 


a  -+  &,i  (*) 


r»  r\  d^,t  (x) 

^  (x)  - 

for  its  derivative.  Then  D  is  the  solution  of  the  linearized  equation 


dD  =  ft(t,  £)Ddt 


with  Ds>s  =  I,  the  d  x  d  identity  matrix. 
In  integrated  form  this  is 


Ds>t  -  I  +  J  Mu,  6, « [x))DSiU  du. 


Consider  the  matrix  Vltt  defined  by 


v.,l  =/-  f  V,,„ /{(«,?,,„  (*))<*» 

J  8 


i.e.,  dV  =  —  V f^{t,i)dt.  Then  VSigD3iS  =  /  and  from  (2.3)  and  (2.5) 


d(V2?)  =  (d7)£>  +  V(d27) 


=  —V  f^Ddt  +  V  f^Ddt 


so  V$tt  Da,t  =  I  for  all  t  >  0.  Consequently, 


.  Kl  ,  (OM)" . 


Suppose  zt,  for  t  >  s,  is  some  differentiable  path  in  Rd.  Rather  than  the  map  x  — >  Cs,t  (x) 
we  can  consider  the  composite  map 


zt  -*■  (**)• 


2 


Note  t  occurs  twice  on  the  right,  so  taking  the  derivative  we  have: 

is,t  (zt)  =  ze  +  f  /(«,  £9(U  [zu))dn 

J 6 

+  J8  {zu)dZu‘ 

Suppose  g:Rd  ->Rd  is  a  second  function  like  /.  Consider  the  equation 

^6  =  g(t ,  £)<&• 

Ihe  unique  solution  of  (2.7)  starting  at  x  e  Rd  at  time  s  will  be  denoted  by  (z), 

tf.t  (x)  ~  z  +  9(u>  (*))<*«• 

We  then  have  the  following  formula  for  £g. 

Theorem  2.1.  (*)  =  ^  (*)  where 

[l  /d£au(zu)\~l  , 

Zt~x+L  <*«))  -  /(«.  et.  (*»)*>. 


PROOF.  As  above,  we  shall  write  (*)  for  ([,  (i)  etc.  Then,  substituting  this 


(2.6): 


£«,<  (^)  —x+f  f{u ,  £8)U  (zu))du 
J  6 

i_  f  (d£a  ,v(^u)\  /$£«,«  (zu)  \  _I  , 

+ Js  v  dz  )  **•»  m  ~  /(w’  &,«  (^«)))^ 

~X  +  J  $(«»  £«,*(*«))**• 


However,  (2.8)  has  a  unique  solution  so 


(x)  =  (^). 

This  result  is  particularly  useful  in  optimal  control  when  one  wishes  to  compute  the  vari¬ 
ation  m  the  cost  due  to  a  perturbation  of  an  optimal  control.  See  [1]  and  (4). 
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3.  BACKWARD  EQUATIONS. 

Consider  the  solution  &,*(#)  of  (2.2).  If  F  :  Rd  — ►  R  is  a  C2  function,  by  the  chain 

rule 

F(t„  W)  =  F(x)  +  J F£(6,«  W)/(u,f„.  (*))*..  (3.1) 

Consider  a  partition  7T  =  (s  =  to  <  *1  <  •  •  •  <  tn  =  t}  of  [s,  f]  and  write 
|tt|  =  max  — 1{\.  Then  one  can  also  write 

t 

(*))  -  f(*)  =  £  (F((tt,i(x))  -  ,*(*)))•  (3.2) 

k=0 

If  5  <  r  <  t,  by  the  uniqueness  of  the  solutions  of  (2.2),  we  have  the  semigroup  property 
of  the  flows 

f.,1  (*)=Mf.,rM).  (3-3) 

For  the  A;-th  term  in  the  sum  (3.2): 

-  f(fw  M) = k+l  1 1  [£tk,tk+ 1  (X)))  F{£tk+ 1  ,1  (X)) 

=mk+1M)-F(ttk+1)t(x)) 
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where  Ji?*]  <  C(tk+ 1  —  f*)2  <  C\n\{tk+\  —  tk)  for  some  uniform  bound  C.  The  left  side 
of  (3.2)  does  not  involve  the  partition  tt;  considering  partitions  such  that  |7r)  — >  0  we  have 
the  following  ‘backward’  equation: 

F[Ut  (*))  =  F(x)  +  J  FeiZujt  (x))  f{u,x)du.  (3.4) 

Clearly  (3.4)  holds  for  vector  functions  F.  The  solution  (x)  of  (2.1)  is  the  ‘forward’ 
flow  from  x.  Taking  F  :  Rd  — *  Rd  to  be  the  identity,  F(£)  =  £,  we  have  from  (3.4)  the 


following  ‘backward’  equation  for  the  ‘forward’  flow: 


»«,t  (®)  =  x  +  J 


*  d£u,t  (x) 


f(u,x)du. 


By  analogy  with  (2.1)  we  can  also  consider  the  following  ‘backward’  equation: 

-dlt  =-/(»,{.,<)<*»  (3-6) 

A 

with  a  terminal  condition  x  at  time  t.  That  is,  we  consider  the  ‘backward’  process  £s,t  (x) 


defined  by 


Ljt  (x)  =  x  -  J  /(u,  £U)t  (x))du 


so  £t,t  (x)  =  x.  Again,  the  map  x  — ►  £s,t  (x)  is  differentiable  with  a  derivative  — For 
a  smooth  bounded  function  F  :  Rd  — >  R  the  chain  rule,  (in  the  s  variable),  gives: 

nLt (*))  =  F(x)  -  j  F((L,, (*))/(«, L,t (*))<*«.  (3-8) 

This  is  the  ‘backward’  equation  for  the  ‘backward’  flow.  Similarly  to  (3.4)  we  can  establish 
the  ‘forward’  equation  for  the  ‘backward’  flow: 

F{L,t  (x))  -  F[X)  -  I  f({<;s,uKX))  — j{u,x)du .  (3.9) 


In  particular,  taking  F  to  be  the  identity  map  on  Rd ,  F(£)  =  £, 


(x)  =  x  -  J 


d£S)V  (x) 


f(u,x)du. 


(3.10) 


Approximation  arguments,  for  example,  would  tell  us  that 


6,t  (6,«  (*))  =  Ut  (&,<  (s))  =  x- 

However,  let  us  proceed  as  follows:  Differentiating  (3.7)  the  backward  equation  for  the 

A 

pk  t  A 

backward  derivative  ~fit  =  18 

Dtit  ~  I  —  j  f({u,  [x))DUtt  du.  (3.11) 

Consider  the  augmented  flow  defined  by  the  pair  of  equations  (3.7),  (3.11),  with  the  ‘vari¬ 
able’  terminal  condition  Dt,t  =  D.  That  is,  consider  (x)  defined  by  (3.7)  and  DStt  (x,  D) 


defined  by 


D,,t  (x,D)  =  D-J  /{ (u,  (x))DVlt  (i,  D)du. 


(3.12) 


Note  equations  (3.11)  and  (3.12)  are  linear,  so  that,  if  DSit  =  Ds,t{x,I)  =  DS)t{x )  is  the 
unique  solution  of  (3.11),  then  Ds>t(x)D  =  Dtit(x,D )  is  the  unique  solution  of  (3.12). 


Therefore, 


dDSit{x,D)  * 

- dD - D,t  (l)' 


(3.13) 


Applying  (3.10)  to  DStt(x,D),  and  noting  derivatives  in  both  variables  x  and  D  are 
involved,  the  forward  equation  for  Dgit  (x,  D)  is 

DStt{x,D)=D-  J 1  di>*'Ud{*,D)  f{u,x)du 


rt  dbSiU{x,D) 


f{(u,x)du. 


Putting  D  =  I  and  substituting  (3.13),  the  forward  equation  for  Ds>t  (x,  J)  =  i)s,t  ix)  is: 

(x)  =  I  ~  J*  -----  f(u,  x)du-  I*  Ds>u  (x)/f  (u,  x)du. 

Using  (2.6)  we  have 

t  A 

t>.,t  ((.,i  (*))  =  1-1  d-^rL  (*))/(«,  M)du 

Jg  OX  . 


-  J'  bg,u  (6,tt  {x))fi (u,  {x))du  +  J  ~faL  ^>u  (x))du 

ft  A 

-  J  A,«  (£»,u  (*))/«(«  >  £&jV  (x))du. 
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Therefore,  DS)t  (£S)t  (x))  satisfies  the  same  equation,  (2.5),  as  that  defining  VSit .  By  unique¬ 
ness,  we  have  the  following  result: 

Theorem  3.1.  Da,t  (&,<  (x))  =  Va,t  =  D~}  (x). 

Similar  arguments  in  the  opposite  time  direction  applied  to  Da>t  (x),  defined  by  (2.4), 
show  that 

D„(l,(x))  =  b;}(x).  (3.14) 

Using  flows  the  following  result  can  be  established: 

Theorem  3.2.  &,< (&>t (x))  =  2. 

PROOF.  Applying  (2.6),  and  using  the  forward  equation  (3.10)  for  C»,t{x) 

£s,t  (£s,t  (^O)  =  x  +  J  /(tt,  £a,ti  (£«,«  [x)))du 

—  J  Da,v{£a,u[x))Ds,u[x)f{U)X)du 

-  x  +  J  (/(u,  6,«  (6,«  (*)))  ~  /(«,  x))du  (3.15) 

by  (3.14).  However,  Zs,t  (£«,t  (x))  =  x  is  a  solution  of  (3.15),  so  the  result  follows  because 
(3.15)  has  a  unique  solution. 

Similar  arguments  again  show  £4|t  ({s,t  (x))  =  x. 
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Abstract:  Strong  variations  are  described  for  the  e-optimal  control  of  a  class  of  eontroi  problems  for  systems  described  by  stochastic 
diffusion  equations.  The  differentiation  process  developed  identifies  the  adjoint  process. 

Keywords:  Stochastic  control;  stochastic  differential  equations;  diffusion  equation;  minimum  principle. 


1.  Introduction 

In  an  earlier  paper  [4]  the  authors  have  applied  a  powerful  non-convex  minimization  result  established 
by  Ekeland  [3]  to  derive  a  local  approximate  minimum  principle  for  a  partially  observed  control  problem. 
A  metric  is  introduced  on  the  space  of  admissible  controls  which  measures  the  distance  between  two 
controls.  A  strong  variation  of  an  e-optimal  control  leads  to  an  inequality,  where  on  the  right  hand  side 
this  metric  appears.  As  it  can  be  expressed  by  a  measure  on  the  set  where  the  perturbed  e-optimal  control 
and  the  control  itself  differ,  the  inequality  may  be  differentiated  to  obtain  the  local  minimum  principle. 

In  [lj  theorems  by  Bismut  [2]  and  Kunita  [6]  on  stochastic  flows  are  applied  to  give  an  easy  and  explicit 
calculation  of  the  change  in  the  cost  due  to  the  strong  variation  of  an  optimal  control.  These  results  are 
used  here  to  describe  the  strong  variation  of  an  e-optimal  control.  Using  the  result  of  Ekeland  [3]  the 
resulting  inequality  can  be  differentiated,  so  giving  a  completely  new  proof  of  the  results  in  [4). 
Furthermore,  the  differentiation  process  identifies  the  adjoint  process;  this  is  the  main  contribution  of  this 
paper.  The  underlying  model  here  is  the  one  considered  in  [1]  and  differs  from  the  one  in  [4]:  the  drift 
coefficients  /,  It  and  the  diffusion  coefficient  g  in  both  signal  and  observation  process  depend  only  on  the 
current  state  of  the  system,  and  not  on  the  whole  past  as  in  [4j;  the  controls,  however,  need  not  be  Markov. 
Furthermore,  we  impose  differentiability  assumptions  on  these  coefficients  and  on  the  cost  functional. 

To  apply  Ekeland’s  result  we  show  that  the  cost  function  of  the  control  problem  described  in  Section  2 
is  continuous  when  the  control  functions  are  topologized  using  the  metric 

</(«!,  «l)  «/({(/,  X)  e  [0,1]  X  C([0,1],  R")  | !/,(/,  x)  *  «,(/,  -c)}).  (1.1) 

Here  P  is  the  product  of  Lebesgue  measure  on  [0,1]  and  Wiener  measure  on  C([0,1],  R"1). 

Then  Ekeland’s  result  [3]  tells  us  that  for  any  e  >  0  there  is  a  control  function  uf  such  that 

J(ut)  £  inf  j(u)  +  e  (1.2) 

and  for  ail  other  control  functions  u, 

J(u)  ZJ(ue)  -  u).  (1.3) 
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That  is,  ue  minimizes  the  functional 

Jc(ti)  =J(u)  +  ed(u,  uc).  (1.4) 

It  is  then  shown  that  ue  minimizes  to  within  £  the  conditional  expectation  of  a  certain  Hamiltonian 
H(s,  x,  £f,  p,  it),  where  is  the  output  of  the  system  for  control  it,.,  and  p  is  an  adjoint  process  which  will 
be  derived  explicitly. 


2.  The  control  problem 

We  shall  treat  the  same  control  problem  as  in  (1).  Let  as  quickly  sketch  its  basic  properties. 

Suppose  the  staie  of  the  system  is  described  by  the  stochastic  differential  equation 

dWMi.«)d/  +  *(f.€,)dw,.  Z0  =  x0eR‘',  0</<;l.  (2.1) 

We  shall  make  the  following  assumptions: 

Al.  .\y  is  given;  if  Ay  is  a  random  variable  and  P0  its  distribution  the  situation  when  /  |  a  | q  dP0  <  cc  for 
some  q>  n  +  1  can  be  treated  by  including  an  extra  integration  with  respect  to  P0. 

A2.  |.[0.1]xR'/x  U  ->  R lt  is  Borel  measurable,  continuous  on  the  compact  metric  space  U  for  each 
( t ,  a),  continuously  differentiable  in  a  with  derivative  fx,  and  for  some  constant  Ku 

(1+|A|)_,|  /(/.*.  «)|  +  |/v(/,  A,  M )  |  <,KX. 

A3,  g.  [0.1]  x  R‘7  -» R"  ®  R”  is  a  matrix  valued  function.  Borel  measurable,  continuously  differentiable  in 
a  with  derivative  gv,  and  for  some  constant  K2, 

U(L  *)l  +  l  8*(<-  x)\<,K2. 

The  observation  process  is  given  by 

dj-,  =  /'(!,)  d/  +  d»'(.  y,Gh"\  y0  =  0,  0</<l.  (2.2) 

In  the  above  equations  w  =  (n*1 . u")  and  v  =  (v] _ ,»''")  are  independent  Brownian  notions.  We 

also  assume: 

A4.  h:Ud  ->R"'  is  Borel  measurable,  continuously  differentiable  in  a,  and  for  some  constant  K3, 

\h{t,  a)  |  +  \hx(l.  a)  |  <Ky 

As  noted  in  [1]  these  conditions  can  be  relaxed. 

Let  P  denote  Wiener  measure  on  C([0,1],  R")  and  p  denote  Wiener  measure  on  C([0,1),  R"').  Consider 
the  basic  space  S2  =  C([0,1],  R”)  X  C([0,1],  Rm)  and  define  Wiener  measure  P  on  Q  by 

P(du>,  d_y)  =  P(dw)p(d}’). 

The  control  parameter  u  will  take  values  in  a  compact  subset  V  of  some  Euclidean  space  Rk .  Let 
Y=  {}’,}  be  the  right  continuous,  complete  filtration  generated  by  (>,.  a  <  t).  Then  an  admissible  control 
is  a  mapping 

u:  [0,1]  x  c([o,i],  r )  -» u 

which  is  7-predictable.  Write  °U  for  the  set  of  admissible  controls. 
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For  and  jc  e  R'7  write  £",(x)  for  the  strong  solution  of  (2.1)  corresponding  to  control  u  and  such 
that  £“,(.*)  ~  x •  Put 

Z“,(x)  =  exp^J)i(^r(x)Y  dyr-\J‘li(£“r(x))2  dr  j 

and  define  a  new  probability  measure  P"  on  Q  by  dP"/dP  =  Zo,(.x).  Then  under  P"($q,(x0),  yt)  is  a 
solution  of  (2.1)  and  (2.2),  that  is  $H_,(x0)  remains  a  strong  solution  of  (2.1)  and  there  is  an  independent 
Brownian  motion  v  such  that  y,  satisfies  (2.2). 

We  shall  consider  a  terminal  cost  given  by  some  continuously  differentiable,  bounded  function 
c(£o,i(-*o))-  The  cost  for  admissible  control  ue®  then  is 

y(n)  =  £„[c(^,(.v0))]  =E[Zi;A(x0)c(^(x0))}.  (2.3) 

It  is  shown  in  [A]  that  J(u )  is  continuous  on  <1U,  when  is  given  the  topology  induced  by  the  metric 

</(«.,  «a) -£({('•  p)  s  [0,1]  X  C([0,1],  R"‘)  |  «i(/,  y)  +  u20,  y))). 

Furthermore,  it  is  shown  there  that  (<%,  el)  is  a  complete  metric  space. 


3.  Stochastic  flows 


It  is  not  known  whether  there  is  always  an  optimal  control  for  the  problem  described  in  Section  2. 
However,  there  is  always  an  e-optimal  control  for  any  £  >  0.  Consider  such  an  e-optimal  control  ur(t.  n ) 
satisfying  (1.2)  and  (1.3).  From  our  observations  in  [1]  there  exists  a  countable,  dense  subset  of  and  for 
the  strong  variations  below  we  only  take  elements  from  this  subset. 

Let  lift  be  a  strong  variation  of  tte  defined  by: 


jte(t.y)  i(  s  <, ,  <  s  +  h  <] . 
y)  otherwise. 


For  notational  convenience  drop  the  subscript  h  in  u,r  and  let  (£'  ,)  be  the  strong  solution  of  the  dynamics 
(2.1)  with  input  ue,  and  (£“,)  the  solution  for  input  u,  i.e.. 


^:l  =  ^sAx)=x  +  ^f(r,  $lr(x),  ii, (r ))  dr+  J‘g(r,  &r(*))  d 


(3.1) 


and 


=  x  +  f/(r,  €“,(*),  «(r))  dr  +  fg(r.  £,(*))  d»V 


(3.2) 


It  is  well  known  (see  [6])  that  there  are  versions  of  these  solutions  such  that  |'((-);R'/->  R'7  is  almost 
surely  a  diffeomorphism,  with  a  Jacobian  C,  =  (3£*  ,/elx)(x)  which  is  the  solution  of  the  equation 

d 

dC,  =/,(/,  &,(*)>  uc)C,  d/+  Es.Ur-  €.,(x))C,  du'J,  Cs  =  1  (nXn  identity  mat.  (3.3) 

(=■1 


Here  fx  denotes  the  partial  derivative  of  /  with  respect  to  x  and  g[  denotes  the  gradient  of  the  /-th  column 
of  g. 

The  inverse  of  the  Jacobian  D,  =  (9£',(x)/<bc)  ',  C,D,  =  /  for  t  >  s,  also  satisfies  a  stochastic  differen¬ 
tial  matrix  equation,  namely 


(3.4) 
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Next  we  describe  a  change  of  drift  induced  by  considering  a  certain  semimartingale  (z,)  as  the  initial 
condition  in  the  dynamics  equation.  This  result  is  due  to  Bismut  [2],  It  allows  us  to  consider  £",(x)  as  the 
unperturbed  process  ,  with  initial  condition  (z(). 

Theorem  3.1.  For  the  perturbation  u  of  uc  consider  the  semimartingale 


Z,  =  X+  l' {  ^-r(Zr)>  Ur)~f(r’  ts.r(Zr),  «r))  dl\ 


Then  the  process  £',(z,)  is  indistinguishable  from  £“,(*). 

Proof.  From  the  results  of  Bisrnut  ([2],  Theorem  3.1)  £*,(*/)  is  a  semimartingale  with  the  representation 

£.l(2r)>  «')  dr 

.  r‘(WsAXr)\i*?sAZr)\~1/  ,/  *r  /  \  \  tt  rr  t  \  r\\  i 

- II - 3x - J  M  ’  &r(*r).  «r)  “/(''•  £.r(2r)>  «j)  df 


+  J‘s(r>  £.r(Jr))  d‘V 


That  is, 


&i(*»)**  +  ff(r’  ZlAzr)*  «r)  dr  +  fg(r.  £r(z,))  duV. 


As  the  solution  (£'',(x))  of  (3.2)  is  unique  it  follows  tha  £',(z,)  =  £“,(.v).  Note  that  for  />s  + 
/»,  1^(2,)  =  IL(2^a)  =  £+t,A£s.s  *•/.(*))•  because  u  equals  uc  tz  s  +  h  and  so  z,  =  zs for  /  >  s  +  h. 
□ 

Next  we  study  the  augmented  flow  (£esl(x),  Z‘,(x,  z)),  i.e.,  the  diffeomorphism  on  R‘/(  1  given  by 
£.,(*)=-*  +  fj{r,  £.r(x),  ur)  dr  +  jfg(r,  £.,(*))  d»v, 

Z!A*)  =  2  +  fz;A*'  A  it  As  A*))'  d  yr.  (3.7) 

Js 

To  justify  this  discussion  note  there  is  a  strong  solution  of  (3.7)  defined  for  every  «£(/  and  y  e  C([0,1], 
R"),  because  the  stochastic  integral  in  the  exponential  defining  Z  can  be  integrated  by  parts. 

As  3 £cs  t(x)/d:  =  0,  3//3r  =  0,  3g/3r  =  0,  the  Jacobian  for  the  augmented  flow  may  be  represented  as 

r  KM  Q 

C  =  *  s  a7t  (  ,  (3.8) 

3ZsAx)  dZ,Ax,  z) 

3;c  3z 


and  the  Jacobian  of  Z' ,  satisfies 

s#ii-  £  jT^o 


-  ZjfUx,  •-)  M%(j0)  +KC,(r))3Z;'3(t*'  ‘->  dr/ 


for  1  ^  ^  d.  Here  summation  takes  place  over  double  indices. 

Obviously,  we  are  only  interested  in  the  solution  of  (3.9)  for  the  case  z  =  1.  Write  Z5,( x)  —  Zf,(x,  1) 
and  from  [1]  we  cite  the  following  result. 
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Lemma  3.2.  (i)  The  following  representation  holds: 


3ZM-V)  .  7e 
a.v 


3  £,(*) 


and  (ii)  Z*,(zt)  =  Z‘‘,(x),  where  (z,)  is  the  semimartingale  defined  in  (3.5). 
Again  note  here  for  t  >  s  +  h, 

z;a*,)  =  z;., (*,+*)• 


4.  Tlie  minimum  principle 

The  cost  associated  with  tT  is  given  by 
y(Mf)  =  £[%(.x0)c(^1(.v0))] 

=  E\Z^S(  x0  )  z;.i  ( .V )  c(  ( X ))]  (where  = 
and  the  cost  corresponding  to  the  perturbed  control  is 

y(«)  =  £[z5.,(*0)z;.I(JC)c(i“I(*))]-£[z5.,(x0)z;.I(rI+*)c(€;.I(.-„*))]. 

Now  Z'j(-)  and  t (!'  ,(  -))  are  continuously  differentiable,  so  we  can  compute  the  difference  between  these 
costs  as: 

J(u‘)  -J(u)  =  E  f  A(s.  r.  zr)(/(r.  £.r(zr).  «;)“/('•.  £.,(*,).  «,))  dr 


where 


A(j.  r,  z,)  =  Z^Z'^z,)! c,(£.,(zr)) 


,3|Ll M 


/(K*)-y(«i)=  r+,£[A(z.  r.  zr)-A(z,  r,  *)](/(r,  £,(*,)•  u')-j(r,  £,(*,).  «,))  dr 

JS 

+  /,+*£[A(j.  r.  .v)-A(r,  r,  .v)](/(r,  £.,(->)<  «;) -/(r.  &r.  «,))  dr 
+  rV[A(r.  r.  .x)( /"(r,  £r(zr).  «;)-/(r.  f'r(z,),  «,) 

Js 

-fir,  £.,(■*)«  «')+/('■>  &r(*)»  «r))]  dr 
+  f  *^ [ A *)(/('■>  £L-(*o)>  »■ ‘r)-f(r ,  lo.,(-v0).  »r))l  dr 


=  /,(/*)  +h(h)  +  ji(h)  +  l4(h),  say. 
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r,  zr)-A(s,  r,  x)  |(l  +  ||ft .U)!Uj,)]  dr 


<,  Kih  sup 

F[jA(s,  /•. 

*r) 

-  A(s,  r. 

*)l(i+l|£o..(*. 

ssr£s+h 

hVl)  1 

1  £  K2h  sup 

E\ |A(s,  r. 

-v) 

—  A(r,  r, 

*)l(l+l|£o..(*< 

s£r£  f + // 

W>)\ 

<  K:Ji  sup 

sgr&s+h 

£[|A(r,  r. 

-*) 

-£.r(*)IL/,]- 

The  differences  on  the  right  hand  side  of  the  three  above  inequalities  are  uniformly  bounded  in  some  Lp 
and  they  converge  to  zero  a.s.  when  It  goes  to  zeto.  So,  the  differences  converge  to  zero  in  L;)-norm.  Then 

lint  (/;)  =  0  for  *  =  1,2.  3. 

i/~0 

From  our  remark  in  the  beginning  of  Section  3,  it  is  an  element  in  a  countable  dense  subset  of  all.  As 

f  E[A(r.  x)(f(r .  ^r(x0),  u')-f(r ,  ^r.  «,))]  dr 
■'o 

is  finite,  there  are  null  sets  N(u')  and  N(u),  such  that  for  s  £  N(u)  U  N(ue)  the  above  expression  is 
differentiable,  and 

lim  /i',(;(#') -J{u))  =  E[A{s,  s,  x)(f{s ,  &.*  (*<>)•  “')~f(s-  So.,(*o)-  «,)]• 

/;  —  0  + 

Therefore,  from  (1.3), 

E[A(s.  s.  x)(/(s.  foJx0).  ufs)-J(s.  ioJx 0),  «,))]  S  lim  («/(«,.  «)r)  <e  (4.1) 


because  d(u.  id)  <h.  Using  a  general  result  of  Kushner  [7J.  the  same  relation  holds  for  the  conditional 
expectation  E[  \YS\. 

Write 


Pt(x)  =  E,  ce(t'0A(x0))’^X-  +  c(€§.1(xo))|jf,At(ft.0(jc0)) 

where  Er  denotes  expectation  under  Pu\ 

Then  substituting  in  (4.1)  we  arrive  at  our  minimum  principle: 


invf.v} , 


Theorem  4.1.  Let  ue  be  an  e-optima l  control  and  let  u  be  any  control  in  °U.  Then  there  is  a  Lebesgue  null  set 
N,  such  that  for  s<£  N  the  following  inequality  holds'. 

Et[pt(x){f{s,  £o.c(.*o),  <)-/(*’  £L(*o)<  a-c-  D 


Remark  4.2.  (i)  Note  that  if  an  optimal  control  it*  exists,  and  if  id  is  replaced  by  u*  in  Theorem  4.1,  then 
we  obtain  the  minimum  principle  in  (1  ]. 

(ii)  If  J(u)  is  Gateaux  differentiable  we  may  deduce  from  Theorem  4.1  the  e-minimum  principle  in  [5], 
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1.  Introduction.  The  theory  of  stochastic  flows  was  developed  by  Kunita,  [4],  and 
Bismut,  [2].  The  concepts  and  techniques  from  this  theory  have  been  used  to  discuss  the 
Mailiavin  calculus  [l],  and  have  again  returned  to  deterministic  flows,  [3],  In  this  paper 
we  show  how  concepts  from  the  theory  of  deterministic  flows  can  be  used  to  provide  an 
elegant  proof  of  the  Pontrjagin  minimum  principle. 

2.  The  Dynamics  of  the  Optimal  Control  Problem  as  a  Flow. 

Consider  the  control  problem 

(l)  x  =  /(*,£,  t/),  0  <t<T,  z(t)  6  Rd,  z(0)  =  Xq  ,  with  u(-)  measurable,  u(f)  €  U  C  R"x  ■ 
We  assume  that  /  is  sufficiently  well-behaved  that: 

(a)  solutions  to  initial  value  problems  are  unique  in  [0,  T)  x  Rd  and  each  solution 
extends  to  (0,  Tj; 
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(b)  the  solution  to  the  IV P  (1),  x(s)  =  xo,  is  a  continuously  differentiable  function 
of  Xo. 

Let  the  associated  cost  functional,  to  be  minimized,  be  defined  by: 

(2)  c[x(T)],  c[.)  :  Rd  R  differentiable.  We  remark  that  (a),  (b)  need  only  hold  in  a 
“tube”  of  an  appropriate  sort  about  an  optimal  solution  (u*(-),  £*(•))  of  (1),  (2). 
The  Pontrjagin  principle  states  that  if  («*(•)>  x*(-))  an  optimal  solution  of  (l)  (2), 

then  there  exists  an  absolutely  continuous  p(-)  :  (0,  T]  — *■  Rd  such  that 

(3)  minp(t)  •  f(t,x*(t),v)  =  p(t)  ■  /(i,x*(t),  u*(f))  in  [O.T].  In  fact  p(-)  is  a  solution 
of  the  adjoint  equation  to  the  linearization  of  (l)  about  £*(•). 

We  will  show  how  this  principle  follows  naturally  from  the  use  of  ideas  from  the 
theory  of  deterministic  flows. 

If  an  initial  instant  s  €  [0,  T*J,  an  initial  value  x  6  Rn  and  a  control  u(-)  are  given,  we 
write  the  solution  of  (l)  satisfying  x(s)  =  x  as  £“(  (x).  Here  the  superscript  u  indicates  the 
dependence  on  the  choice  of  control  u(-);  in  addition  we  write  ut  for  u(t).  If  (u*(-),  x’(-)) 
is  optimal,  we  write  this  pair  as  (tq* ,  (q  (  (xo))  and 

£o ,t  ( xo )  =  x0  +  /  /(r,  (zo),  K)dr- 

Jo 

For  any  x  E  Rd  and  any  s  €  [0,  T\  we  now  define  t  (x)  by  the  integral  equation: 

(4)  V,jt  (x)  =  x  +  J  f(r,C,Ax)>ur)dr- 


Notice  that  Ct,t  (x)  solves  (l)  as  a  function  of  t,  and  takes  on  the  value  x  at  i  =  s,  but  it  is 
not  necessarily  optimal,  (unless  s  =  0,  x  —  xo).  Our  assumptions  Imply  that  (x)  is  a 
continuously  differentiable  function  of  x.  Differentiating  (4)  with  respect  to  x.  we  obtain 
the  integral  equation  defining  the  matrix  DSit  (x)  =  £r{CSlt  (x)): 

(s)  D,t  (x )  =  /  +  f  f, (r,  (x),  u-r)Dv  (x)dr. 
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We  now  define  Vl>t  (x)  for  any  x  6  Rd  and  any  s,t  in  [0,7’]  by  the  linear  integral 
equation: 


(6) 


V,jt  {x)  =  I-  J  V4>r  (: x)f{r ,  Ct,r  (x)tu*)dr. 


Lemma  1 .  (from  [3]).  V3it  ( x)D,tt  (x)  =  I  for  all  s ,  t  in  [0,  T],  and  all  x  €  Rd . 
PROOF:  Using  (5)  and  (6),  we  see  that 


0) 

00 


7,i4(x)Dm(x)  =  J 

=  -P4(t  (*)/(<,  (*), (*)  +  V«  (*)/(*.  (*W  (*) 


=  0. 

Thus,  P8i(  (x)  =  [Z)gii  (a:)]-1  for  all  s,t  in  (0,7’j  and  all  x  E  Rd.  In  particular,  we  conclude 
that  DSit  (x)  is  always  invertible. 

Next,  for  any  continuous  map  (path)  zt  :  [0,T]  — *•  Rd,  we  consider  the  composite 
map  <p  :  f  ►->  ( zt ),  i.e.,  the  function  <£(t)  =  £s,t[zt)  defined  for  s  and  t  in  (0,T]  by  the 

integral  equation: 


(7) 


4>[t)  =  z,  + 


u(r))dr  + 


This  equation  is  obtained  from  (4)  by  differentiation  with  respect  to  t. 

We  now  perturb  the  given  optimal  control  u*  (•)  in  the  by  now  standard  manner: 


(8) 


uf  = 


outside  [s,s  +  h], 


u  €  V  inside  [s,s  +  h), 
and  define  the  curve  zt  :  (0,  T]  — *  Rd  by  the  somewhat  improbable  integral  equation  (note 
that  the  subscript  (s,r)  is  reversed  from  (4)): 


Cn\ 


=x+  y  [D',rW)\~X  [f{r,C,r(zr),Ur)  -  f{r,  (zT) ,  uT)]dr. 


Lemma  2. 


PROOF:  By  (7)  and  (9), 


C,t  {zt)=x  +  j  f(r,  ( zr)lu*r)dr+ 

+  i  D»,r{zr)[Dttr{zrj]~l  [/(r,  Ca>t  (zr),  ur)  -  /(r,  ^  (zr),  Ojdr 

=  X  +  L  f(r't*e>t(Zr),Ur)dr‘ 


The  assertion  follows  from  the  uniqueness  of  solutions  to  (l). 


3.  The  Minimum  Principle.  If  we  define  x  —  (xo),  for  a  given  s  6  [0,T],  then 

the  optimal  cost  can  be  written 

c|*fl  =  c[fo,r  (zo)]  =  c[C+h,T  {C,i+h  (*))], 

for  any  h  >  0,  s  G  [0,  T],  s  +  h  6  [0,  T],  with  x  =  £o,»  (zo)- 
Since  u(-)  cannot  give  a  lower  cost  than  u*(-), 


(*))]  <  c[Cffc,T  {C,  s+k  (*))]  ^ 

—  C[^«+A,r  {ts,s+h  [zs+h.  ))]i 


c[^V(x)]  -cU*V(z'+*)]  - 0 


for  all  a  €  (0,TJ,  h  >  0,  (s  +  h)  6  [O.T]. 

This,  with  judicious  application  of  Mean  Value  Theorems  and  letting  h  tend  to  zero, 
leads  to  the  Pontrjagin  principle.  In  fact,  by  the  Mean- Value  Theorem, 


■ 

~q7  '  (C,T  ( x )  "  &,T  (zs+h )]  <  0, 
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for  s,h  as  above,  where  the  gradient  of  c(-)  is  evaluated  somewhere  on  the  line  between 
Z*tT  (x)  and  £*j.  (z«4*  )  in  Rd.  Since  we  will  shortly  let  h  decrease  to  zero,  this  evaluation 
point  will  become  £*r  (x)  =  xf . 

It  follows  from  (4)  and  our  assumptions  on  /  that 

(11)  -•**+*]  <  °, 

but  here  the  rows  of  Dtir  =  d£a,T  ( x)/dx  are  evaluated  at  perhaps  different  points  between 
x  and  ,  because  the  Mean- Value  Theorem  is  only  valid  for  real-valued  mappings  and 
hence  must  be  applied  to  each  component  of  .  From  (9)  we  can  write 

/*+h 

[Dt,r  (z^]"1  {/(r,  C>r  (zr),  K)  ~  f{r>  ta,r  {zr),UT\dr. 

Combining  (11)  and  (12),  dividing  by  h  >  0  and  letting  h  go  to  zero,  we  obtain  (noting 
that  D„  =  I,  z,  -  x  =  £,*  ,): 

0  a  (4)]  D.,r  )  ■  i/(*, )  -  /(*, ,5)1. 

This  is  the  Pontrjagin  principle  (3)  with 


'>M  =  [|f(4)]^f(C). 


4.  The  adjoint  equation. 

By  the  semigroup  property  of  the  solution  flows,  for  0  <  s  <  t  <  T\ 

(!■*)  (*.)  =  «,,(«,  .M- 

a  .** 

Writing  D'Sit  =  2^(  x)  and  differentiating  (14)  by  the  chain  rule 


Bo,t  ~  u$,t  (to,*  (*o))L>o,s  (xo)- 


From  (13) 


M  =  §|(4)0.‘,r  (€,.)■ 
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Therefore,  using  (15) 


(16) 


p(s)B o,.(*o)  =  rMr)D-x(Xo)  =  constant. 


Differentiating  (16)  in  s 

pdD*  4-  {dp)D*  =  0. 

That  is  dp  =  (— pdD'jD*"1 .  (We  have  noted  in  Lemma  1  that  Dq'1  =  V0*g  exists.) 
From  (5) 

dDolS  =/*(«,  £o,.(*o),  u*)Do,a^- 
Therefore,  p(s)  is  the  solution  of  the  equation 

dp(s)  =  -p(s)/a(s,  ^  (xo),  u*e)ds 


with  initial  condition 

P(°)  =  (4)^0, T- 
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1.  Introduction. 

From  a  very  simple  representation  of  the  integrand  in  the  integral  representation  of  a 
martingale,  we  derive  an  integration  by  parts  formula.  This  is  used  to  give  a  new  proof  of 
the  existence  of  a  density  of  a  diffusion  process  under  the  hypothesis  that  the  inverse  of 
the  Malliavin  matrix  is  in  some  T^-space,  a  result  implied  ’  _  Ilormander’s  condition  Hi. 

Following  Malliavin’s  original  proof  of  this  result  there  have  been  other  approaches  to 
what  is  now  known  as  Malliavin’s  calculus,  including  those  of  Stroock  [17],  Shigekawa  [1G), 
Bismut  [4j,  Bichteler  and  Fonken  [2],  and  Norris  [15].  The  main  simplification  in  this 
paper  is  the  observation  that  no  infinite  dimensional  calculus  of  variations  is  required. 
This  calculus  can  be  replaced  by  ordinary  differentiation  in  finite  dimensional  spaces. 
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2.  Some  history  and  the  Hi  condition. 

Let  us  consider  the  unique  solution  £s,t(x)  of  the  stochastic  differential  equation 

d£s,t(x)  =  Xa(t,£3jt(x))dt  +  Xi(t,^3it(x))dw't 

=  (1) 

where  ( wt )  =  (w}, . . . ,  w™)  is  an  m-dimensional  Brownian  motion  on  (f2,  X,  Xt,  P)  and 
Aro,  A'-!, . . . , Xm  are  smooth  vector  fields  on  [0,  oo)  x  IR^,  all  of  whose  derivatives  are 
bounded. 

It  is  a  well  known  fact  from  harmonic  analysis  that  £o,t(x)  has  a  density  if 

l-Ec4(£0>r(zo))l  <  K  sup  |c(i)|,  (2) 

rent* 

where  c  is  any  bounded,  smooth  function  with  bounded  derivatives  [14,17,4,20].  Using 
different  methods  Malliavin  [14],  Stroock  [17],  Shigekawa  [16],  and  Bismut  [4]  showed  that 
(2)  is  true  if  the  inverse  of  the  Malliavin  matrix  Mot  is  in  some  Lp(£l),  and  they  linked 
this  result  with  Hormander’s  famous  result  to  conclude  that  is  in  all  Lp(ft),  p  <  oo, 
if  Hormander’s  condition  Hi  is  satisfied: 

Condition  Hi:  Aj,...,Am,  [A,',A;-],  [A,-, [Ay,  A*]], . . . ,  ij,  k  =  0, . . . , m  at  xo  span 

]Rd. 

Malliavin’s  approach  is  based  on  a  function  space  martingale  calculus  which  comes 
from  the  Ornstein-Uhlenbeck  process  on  Wiener  space  [14];  this  is  now  known  as  Malli¬ 
avin’s  calculus  of  variations.  Shigekawa  [16]  provided  an  alternative  formulation  which 
relies  on  a  Sobolev-type  extension  of  Frechet  derivatives  with  Wiener  measure  replacing 
the  Lebesgue  measure  in  the  finite  dimensional  situation,  and  he  makes  no  use  at  all 
of  the  Ornstein-Uhlenbeck  process.  Stroock  [17,18]  also  avoids  this  process  in  his  en¬ 
tirely  functional-analytic  reformulation  of  the  Malliavin  calculus.  So  far  the  approaches  of 
Shigekawa  and  Stroock  (also  cf.  Ikeda  and  Watanabe’s  contribution  [12])  are  reformula¬ 
tions  of  Malliavin’s  approach. 

o 


Roughly  speaking,  these  approaches  rely  on  the  analysis  of  a  differential  operator  £, 
which  may  be  seen  on  the  one  hand  as  an  operation  on  the  Wiener  chaos  decomposition 
of  a  Brownian  functioned  F(u>) 

~  rT  fh 

£F(u>)  =  /_,  m  /  •••  /  ...dw(m, 

m  =  l  J° 

or  as  the  generator  of  a  time  changed  Brownian  sheet  { ST(t )  |  (r,  t)  6  (0,oo)2},  namely 

Vr(t)  =  e~^Ser(t) 


seen  as  a  process  on  C(0,  oo).  For  a  “good”  function  c,  we  then  find 

c'(F)  =  |(  -  £(Fc(F))  +  c(F)CF  +  FCF)  •  A~\  (3) 

where  A  is  the  inverse  of  the  Malliavin  matrix  A  =  ( DF,DF )  =  Y(^h‘F)2  and  D^i 

i 

is  the  directional  derivative  in  the  direction  of  the  integrated  element  hl  of  a  complete 
orthonormal  system  on  [0,T].  The  analysis  of  the  right  hand  side  of  (3)  then  leads  to  a 
bound  on  F|c'(F)|  as  required  in  (2). 

Zakai  pointed  out  that  £F  may  also  be  seen  as  the  F2-limit  of 

F(u>)  -  E[F(y/l^e  w  +  s/lw)  \  Fw\ 

€  ’ 

where  the  relation  to  the  generator  of  the  infinite  dimensional  Ornstein-Uhlenbeck  process 
becomes  apparent  [19,20],  as  this  non-coherent  derivative  may  be  interpreted  as 


dF((w) 

3C 


=  D^F  —  trace  D2F. 


(4) 


Bismut  however  gives  a  different  approach  which  expresses  the  Wiener  space  deriva¬ 
tives  as  function  space  derivatives  in  a  Girsanov  functional.  The  basic  idea  here  is  a 
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perturbation  of  Brownian  motion  by  a  small  drift  e  ■  f  usds  (u.,  a  predictable  function). 
Then 

DuF(w)  =  ~  F(w  +  e  J  uds ) 

However, 

F(F(to)|  =  E[F(w  +  £  J  uds)  ■  7j>) 

where  7 t  is  the  Girsanov  functional,  the  solution  of 

A 

7T  =  1  -  £  y  -y3uadwa. 


With 


E[F(w)\  =  E[F{w  +  J  eud,)  ■  7 t\ 

»  F(F(u;)]  +  £F[FuF(w)]  -  eE[F(w)  J  uds } 

we  find  the  Bismut  integration  by  parts  formula 

F[FH  J  usdw3 ]  =  E[DuF(w)}. 

Applying  this  to  “nice”  functions  c(F)  ■  g,  formally  we  find 

E[c(F){DuF)-1  j  udw]  =  E[J(F)(DuF)-lDuF  +  c{F)Du({DnF)-1)] 

and 

Ec'(F)  =  E(c(F)(DuF)~l  j  udw )  -  E(c(F)Du(DllF)~1).  (5) 

( DUF )  now  plays  the  role  of  the  Malliavin  matrix,  and  the  assumption  that  DUF  >  0 
for  a  suitable  predictable  ( us )  leads  to  a  bound  on  |Fc'(F)|  in  (4)  as  required  in  (2). 

In  the  survey  article,  (20],  Zakai  points  out  that  the  Malliavin  and  Bismut  approaches 
are  not  equivalent. 

We  follow  here,  more-or-less,  the  Bismut  approach,  but  where  Bismut  considers  vari¬ 
ations  in  a  function  space  our  formulation  reduces  the  Malliavin  calculus  to  differentiation 
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in  a  finite  dimensional  space  for  the  situation  where  the  Wiener  functional  is  just  a  solu¬ 
tion  of  a  diffusion  equation  as  in  (1),  F(w)  =  £o7’(xo)-  The  key  observation  which  leads  to 
our  result  is  a  martingale  representation  formula  which  might  be  seen  as  coming  from  the 
folklore  of  mathematics,  but  it  provides  us  with  a  new  formulation  of  the  integration  by 
parts  formula,  which  -  as  is  well  known  -  always  plays  the  fundamental  role  in  Malliavin’s 
calculus. 


3.  Representation  of  martingales. 

Consider  the  solution  ^(xo)  of  (1)  and  let  c  be  a  twice  continuously  differentiable 
function  for  which  c^o.rfao))  and  the  components  of  c^fo.r^o))  are  integrable.  We  then 
have  the  following  representation  for  the  right  continuous  version  of  £[c(^o,r(;Eo))  I  Ft]  =: 
mt. 

THEOREM  3.1.  The  martingale  mt,  0  <  t  <  T,  has  a  representation  as 

mt  =  £[c(£0,t(zo))]  +  /  7 i(s)dwla 

Jo 

with 

7 '.■(«)  =  E[c(£o,t(xo))Do,t  I  F^D^X^s,  (6) 


where  DSit  is  the  Jacobian  of  the  stochastic  flow, 


dx 


Note  that  from  the  following  theorem  cited  from  [3,8j  U 3i<  exists  as  a  solution  of  a 
stochastic  differential  equation. 

THEOREM  3.2.  There  is  a  map  £  :  Q  x  (0,  oo)  x  (0,  co)  x  IRd  — »  IR^  such  that 
(i)  for  0  <  s  <  t  <  T,  x  €  IRd,  £a,<(x)  is  the  essentially  unique  solution  of  (1 ); 


5 


(ii)  for  each  u,s,t  the  map  •)  is  C’00(IR'J ,  IRd)  with  a  Jacobi  mi  which  satisfies 


d  X  dX 

dDa^t  =  ~~Q£~{t> £s,t(x))D3,tdt  H  ~Q£-(t,£3tt(x))DSttdw\ 
Ds>s  =  /,  the  identity  matrix ; 


d2i 

(Hi)  the  second  derivative  W.t  =  — ~  satisfies 

ox 2 

dW3tt  =  ^(t^s>t(X))W3<tdt  +  ^i(t^a<t(x))Wa>tdw\ 

qi  %  d^X 

+  ®  D3ttdt  +  —S(t,S,tt(x))D.,t  ®  £.t,du;j 

W,.,  =  0  €  IRrf  0  HiJ  ®  IRrf. 


Proof  of  3.1:  Any  fFt -martingale  (m<)  may  be  represented  as 

mt  =m0  +  /  7 ,(s)dw's 
Jo 

for  a  predictable  integrand  7,-.  As  £0,«(zo)  is  Markov 


m<  =  £[c(£0,r(zo))  |  F(J 

=  £[c(6(r(*))  |  Ft) 

=-V(t,x),  where  x  =  Z0il(x0). 

Then  applying  Ito’s  rule  to  V(t,x),  x  =  Zo,t(x 0)  gives 

V(ti£o,tixo))  =  ^(0,xo)  +  J  +  LV^jds 

[‘  dV 

T  J  $2  >(5t £o,i[:i:o))d/w) 
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with 


□ 

Now  let  u(s )  =  (ui(s), . . . ,  um(s ))  be  a  square  integrabie  predictable  process.  Apply¬ 
ing  the  above  representation  we  find  the  desired  integration  by  parts  formula. 

THEOREM  3.3.  Under  the  above  assumptions  the  following  equality  holds 
'T  T 

E  c(£o,t(xo)  [  u,0)dud  =E  (  E[c^0tT(xo))Do,T\  F3}Do  ]Xt(s)ui(s)ds 
Jo  Jo 

=  E 

by  Fubini’s  theorem. 

In  particular,  putting  u,(.s)  =  (D^Af,(.s))*  and  considering  the  product  function 
WSo,t(x o))  =  c{Zo,T{xo))g(t.o,T(xo)  we  have 

Theorem  3.4. 

fT 

zWoAxoimorixo))]  I  {D^Xi{s)ydiv'3  =  E[{c^g  +  cg^D^M^l 

Jo 

where 

D-'ux{(u)x;(u)D;-ldu. 


cdto,T(xo))D0,T  /  D0  lsXi(s)ui(s)ds 
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Ma>t  is  the  Maliiavin  matrix. 

In  order  to  obtain  a  bound  on  we  now  would  like  to  take 


□ 


0  —  M0  'pT)  o  p, 

but  this  function  not  only  depends  on  <fo ,r-  To  get  around  this  difficulty  we  have  to  enlarge 
the  system  in  the  following  way. 

4.  Existence  of  a  density  for  <fo,r(zo)« 

When  enlarging  the  system  the  results  of  3.2  might  no  longer  hold  for  replaced  by 
the  new  system  as  the  coefficients  are  no  longer  bounded. 

We  consider  the  flow  defined  by  (1),  its  Jacobian  D3i(,  the  martingale 
RSit(x )  =  fJ(D~flX;(u))*dw,u,  and  the  inverse  of  the  Maliiavin  matrix 
M,.,  =  £/,'  D-lXi(u)X:(u)D;-'du.  Then  for 

3,  t,  x)  =  (,,,(x),  x  =  (oAxo) 

D‘°’(x)  =  D,,,(  x),  D  =  D,Axo) 

D™(x0)  =  D,,,D 

<?(  x)  =  J‘(D-lXi(u)ydw-u  (8) 

<  =JJ +  £>-'<, (x),  R  =  K,, 

M?)  =  M.,,(x) 

=  M  +  D-lM,,,(x)D-\  M  =  Ml, 

the  enlarged  system  <f>^  —  (<j>° ,  D°,  i?°,  M°)  is  Markov.  We  now  would  like  to  apply  a  result 
similar  to  3.2  to  this  enlarged  system.  Introduce  the  set  SQ(di , . . . ,  <4),  a,  d,  dt , . . . ,  <4 
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positive  integers,  of  C°°  functions  X  :  IRd  — >  ITT*  of  the  triangular  form 

A'^2^(a;l,a;2)  (  \ 

X(x)  =  for  x  =  |  ;  ,  x‘  e  IRa‘' 


\Xk(xl,x2,. ..  ,xk)J 
and  IR'*  =  HT*1  x  •  •  •  x  ER^ ,  which  satisfy 


Ms(«,N)  =  sup  (  sup  V  sup  IDjX(j)(x)\) 

r€Kla  X0<n<N  1  +  FT  X<;<*  ' 


for  all  N. 


Note  that  <f>*  is  Markov  with  coefficients  in  S(d,  d+a f2,  2 d+cP,  2d+cP),  and  following 
Norris  [15]  we  may  state  the  extension  of  3.2. 

THEOREM  4.1.  Let  x0,xi, . . .  ,xm  €  Sa(di,. . .  ,<4).  Then  there  Is  a  map  <f> :  Six  (0,co)2  x 
— ♦  IR^  such  that 

(i)  for  0  <  s  <  t,  x  €  BT*,  (f>  is  the  essentially  unique  solution  of 

dxt  =  X0  (x)dt  +  Xi(x)dw\,  xs  —  x ; 

(ii )  for  all  (to,  s,  t),  <f>  is  C°°  with  derivatives  of  all  orders  satisfying  stochastic  differential 
equations; 


sup  B  sup  \Dn<f>(u,s,u,x)\p 


<  C(p,R,N,du...,dk,  a,  j|A'o||sa.,v,  •  •  •  >  li^m||sa,N)- 


Furthermore,  we  can  consider  the  Jacobian  of  say  D^\  and  construct  R.[1}  = 


f*  Di]l  1  x\l\u)dw'u,  and  let  M^J  =  {R^J  QR^t*)  the  predictable  quadratic  variation 


rO)  _  /p(J)  /O,  p(°) 


of  Z?1  nr>A  /?o» 


This  4-tuple  defines  <f> ^  7  R^\  M^)  and  inductively  we  can  proceed  to 

define  for  all  n,  and  Norris’  result  holds  for  all  <f>( ri\ 

Now  apply  3.4  to  c(<f>^)  •  to  obtain: 


Corollary  4.2. 

W(0))5W<1))  ®  JI<0>1  =  B!(Voc)(^°)!,(^1)Do,tM„,t] 

+  B(c(<6»)(ViJ)W,)D1M‘]. 

and  for 

«(< *(1)) = 

we  find 

£M£o,r(zo))]  =  ^[c(^o,r(a:o))M0“^D^  ®  RoiT] 

-if[c(^o,r(xo))(Vl5)I>o,rMo1T4^M0(^]. 

□ 

An  application  of  Jensen’s,  Burkholder’s,  and  Gronwall’s  inequalities  with  Norris’ 
result  implies  that  all  terms,  except  possibly  Mq  ?,  are  in  all  I/,  p  <  oo.  If  now  we 
assume  that  is  in  some  Lp,  e.g.,  if  we  assume  HI  to  hold,  then  we  have  the  desired 
result. 

THEOREM  4.3.  Let  £o,t(zo)  be  the  solution  of  (1)  and  c  a  bounded  C°°  function  with 
bounded  derivatives.  Then  if  M0"j’  is  in  some  Lp 

|£M£o,rOo)]|  <  I<  sup  |c(x)|. 

rSIR* 

With  this  result,  D.  Williams’  ‘ridiculous’  example  on  the  existence  of  a  density  for 
the  Brownian  motion  really  becomes  trivial: 

|f/(c'(t<;i)]|  <  \E[c(wi )  •  ioi]|  <  sup  c(x)  •  const. 

x€lR<i 
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5.  Application. 

The  Malliavin  calculus  could  not  have  attracted  so  much  attention  if  there  were 
not  many  important  applications,  together  with  the  remarkable  fact  that  it  links  the 
Hormander  partial  differential  equation  methods  with  probabilistic  aspects.  Within  stochas¬ 
tic  analysis  it  provides  many  helpful  tools,  such  as,  for  example,  the  integration  by  parts 
formula  which  is  equivalent  to  a  martingale  representation  theorem.  In  filtering  theory, 
D.  Michel  and  J.M.  Bismut  [5]  used  the  calculus  of  variations  to  prove  the  existence  of 
densities  for  optimal  filters,  and  Jacod  and  Bichteler  [l]  extended  these  results  to  diffusion 
processes  with  jumps. 

Many  of  these  results  can  be  simplified  by  using  the  finite  dimensional  calculus  devel¬ 
oped  above.  The  full  details  are  found  in  [9,10,11]. 

J.M.  Bismut  [7]  (also  cf.  [13])  applied  the  results  from  Malliavin’s  calculus  to  the 
theory  of  index  theorems  in  algebraic  topology  and  to  large  deviations  problems  [6]. 

Recently,  there  have  been  several  attempts  to  develop  a  notion  of  anticipative  stochas¬ 
tic  integrals.  This  would  allow  one  to  consider  functions  u(s)  above  which  might  not  be 
predictable  and,  in  turn,  this  would  then  allow  the  development  of  Bismut’s  Malliavin 
calculus  to  its  full  strength. 
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Martingale  Representation  and  Hedging  Policies 

David  B.  COLWELL 
Robert  J.  ELLIOTT 
P.  Ekkehard  KOPP* 

Department  of  Statistics  and  Applied  Probability 

University  of  Alberta 

Edmonton,  Alberta,  Canada  T6G  2G1 

The  integrand,  when  a  martingale  under  an  equivalent  measure  is  represented 
as  a  stochastic  integral,  is  determined  by  elementary  methods  in  the  Markov 
situation.  Applications  to  hedging  portfolios  in  finance  are  described. 

martingale  representation  *  Girsanov’s  theorem  *  stochastic  flow  *  diffusion  * 
hedging  portfolio 

1.  Introduction. 

In  the  modern  theory  of  option  pricing  and  hedging,  the  representation  of  martingales 
as  stochastic  integrals  plays  a  central  role.  Since  the  corresponding  integrands  immediately 
lead  to  hedging  strategies,  it  is  of  considerable  interest  to  find  explicit  expressions  for  these 
integrands. 

The  martingale  representation  result  and  its  background  is  fully  described  in  the  paper 
of  Ocone  [12],  where  the  problem  is  discussed  using  methods  of  the  Malliavin  calculus  and 
weak  differentiability  in  certain  Sobolev  spaces.  In  a  recent  paper  by  Ocone  and  Karatzas 
[13]  the  representation  result  of  [12]  is  applied  to  determine  optimal  portfolios  and  hedging 
strategies. 

In  the  Markov  case  elementary  methods,  which  do  not  use  the  Malliavin  calculus  in 
function  space,  are  employed  by  Elliott  and  Kohlmann  in  [5]  and  [6]  to  determine  the 
integrands  in  certain  stochastic  integrals.  Indeed,  all  that  is  used  is  the  Markov  property 
and  the  Ito  differentiation  rule. 

The  present  paper  extends  the  representation  result  of  [6]  to  the  situation  where  the 
martingale  representation  takes  place  with  respect  to  an  equivalent  measure  whose  Gir- 
sanov  exponential  is  defined  in  terms  of  a  Markov  integrand.  The  motivation  for  the 
Girsanov  measure  transformation  is  developed  by  Harrison,  Kreps  and  Pliska  [8],  [9].  A 
Maxkov  Girsanov  transform  clearly  introduces  a  new  integrand  in  the  martingale  represen¬ 
tation,  and  this  is  made  explicit  in  Theorem  3.1.  It  is  possible  this  result  could  be  derived 
as  a  corollary  of  the  general  result  of  Haussman  and  Ocone,  see  [12].  However  any  such 
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relation  is  certainly  not  transparent  and  our  proof,  again  in  the  Markov  case,  is  simple  and 
direct. 

The  application  of  our  martingale  representation  result  to  option  pricing  is  described 
in  Section  4.  Stock  price  dynamics  which  give  explicit,  closed  form  expressions  for  hedging 
policies  appear  hard  to  find.  However,  in  Section  5  we  show  how  our  result  gives  the  hedging 
policy  in  the  well  known  Black-Scholes  case,  [1],  of  log-normal  prices  with  constant  drift 
and  variance. 


2.  Dynamics. 

Suppose  w  —  (tu1, . . . ,  wm )  is  an  m-dimensional  Brownian  motion  defined  for  t  >  0  on 
a  probability  space  (fy.PjP).  Consider  the  d-dimensional  stochastic  differential  equation 

dxt  =  f(t ,  xt)dt  +  o(t,  xt)dwt  (2.1) 

for  t  >  0,  where  /  :  [0,  co)  x  Rd  ->  Rd  and  a  :  [0,  oo)  x  Rd  -»  Rd  <g>  Rm  are  measurable 
functions  which  are  three  times  differentiable  in  x,  and  which,  together  with  their  deriva¬ 
tives,  have  linear  growth  in  x.  Write  £s,t{x)  f°r  the  solution  of  (2.1)  for  t  >  s  having  initial 
condition  £5)5(x)  —  x-  Then  from  the  results  of  Bismut  [2]  or  Kunita  [11]  there  is  a  set 
N  C  £1  of  measure  zero  such  that  for  w  N  there  is  a  version  of  £s,t(x)  which  is  twice 

differentiable  in  x  and  continuous  in  t  and  s. 

d£  t(x) 

Write  Dsj(x)  =  — ^ — -  for  the  Jacobian  of  the  map  x  — >  £5)j(z);  then  it  is  known 
that  D  is  the  solution  of  t£e  linearized  equation 


dDs,tix)  -  fx(t,  xt)D3yt(x)dt  +  a x(t,  xt)Dsj(x)dwt 

with  initial  condition  Ds,s(x)  =  I,  the  d  x  d  identity  matrix.  The  inverse  D~}(x)  exists; 
see  [2]. 

Suppose  g  :  [0,  co)  x  Rd  — ►  Rm  satisfies  similar  conditions  to  those  of  /  and  define 
the  (scalar)  exponential  Ms^(x)  by 


M),<M  =  1  +  J  Ms,T (l)j(r,  is, ,r(x))  ■  dwr 

=  1  +  i  dw*  ■  g(r,£s,r(x))Ms,r(x), 


(2.3) 


where  *  denotes  adjoint  and  •  inner  product  in  Rd. 

Write  {Ft}  for  the  right  continuous  complete  family  of  cr-fields  generated  by  w.  If,  for 
example,  g  further  satisfies  a  linear  growth  condition 


]$r(f,x)!  <  K(1  +  |x|) 


a  new  probability  measure  P  can  be  defined  by  putting 


dP 

dP 


Ft 


Mo,t(xo)- 


2 


Girsanov’s  theorem  then  implies  that  w  is  an  {Ff}  Brownian  motion  under  P  where 


dwt  =  dwt  -  0(t,  €o,f(zo))<&- 


(2.4) 


Let  c  ;  Rd  R  be  &  C2  function^  which,  together  with  its  derivatives,  has  linear 
growth,  and  for  0  <  t  <  T  consider  the  P  martingale 


Nt  =  e[c(£  o,r(*o))  I  Ftl 

Then  from,  for  example,  Theorem  16.22  of  [4]  Nt  has  a  representation  for  0  <  t  <  T  as 

rt 
10 

where  7  is  an  {Ft}  predictable  process  such  that 

rT 


Nt  =Nq+  f  7 sdws, 
JO 


(2.5) 


/  Eh 

Jo 


'sf  ds  <  00. 


3.  Martingale  Representation. 
Theorem  3.1. 


It  —  E 


r T 

J  dw*  •  ^(r,  £o,r(*o))A),r(*o)  *  c(fo,r(*o)) 


+  cz(Zq,t(xo))do,t(xo)  I  Ft  Do,t(x 0)  •  ff(*»£o,*(*o))- 


PROOF.  For  0  <  t  <  T  write  x  =  £o,t(xo)-  By  the  semigroup  property  of  stochastic 
flows,  which  follows  from  the  uniqueness  of  solutions  of  (2.1), 

£o,t(*o)  =  &,r(£o,t(*o))  “•  (3-1) 

Differentiating  (3.1)  we  see 

dO,t(xo)  =  Et,T(x)DQ,t(x  o)-  (3-2) 

Furthermore, 

m0,t(x0)  =  M0,t(x0)Mt,Tix)-  (3-3) 

For  y  €  Rr  define  V(t,y)  =  E[Mf  T(y)c(£t  T(y)%  and  consider  the  martingale 

Nt  =  E[c(£ o;rixo))  I  Ft\ 

=  E[Mqz'(xo)cUo,t(xq))  1  ffi) 

E[Mq^(xq)  I  Ft) 


3 


=  E[Mt,T{x)c{Zt,T{x))  I  Ft] 

=  E[MtiT{x)c(£tiT{x))},  by  the  Markov  property. 

Then  from  Lemma  14.18  of  [4] 

Nt  =  V(t}x). 

We  noted  above  that  &,T(X)  is  twice  differentiable  in  x\  the  differentiability  of 
E[MtyT(x)c(Zt,T(x))}  in  t  can  be  established  by  writing  the  backward  equation  for 
(M<,T(®)>6,jK*))  ^  in  [11]. 

Under  P,  is  given  by  the  equation 


£o,<(xo)  =  *0  +  J  0,S(*o))  +  o’ '  ^(«,40,tf(*0)))^ 

+  /  cr('s.^O,s(a;o))^- 

J  u 

If  V(t^Q  t(xo))  is  expanded  by  the  Ito  rule  we  see 


(3.4) 


V(t,x)  =  V(t,Z0tt(x0))  =  Nt 

=  U(0,xq)  +  (“^"(^O.s^o))  +  LV^s,  ^O^C^o)))  ds 


Here 


L  -  £  (/'  +  £  + 1  £  “y 

Z=1  ;=1  1  i,j= l 


a2 


dx;dx. 


where  a(t,x t-)  —  (ajj(t,Xf))  is  the  matrix  ct<7*.  Now  iVj  is  a  special  semimartingale,  so  the 
decompositions  (2.5)  and  (3.5)  must  be  the  same.  As  there  is  no  bounded  variation  term 
in  (2.5)  we  see  immediately,  similarly  to  [7],  that 


dV 

dt 


(s>(o,s(xo))  +  LV(s,^s(x0)) 


=  0 


with  V(T,x)  =  c(x).  Also 


dV 


75  =  aj(5»^(*o)M«^o,«(*o)). 


4 


However,  £t,T(x)  —  £o,r(xo)  so>  h'°m  the  differentiability  and  linear  growth  of  g : 


dV(ttx) 

dx 


=  E 


Again  using  the  existence  of  solutions  of  stochastic  differential  equations  which  are 
differentiable  in  their  initial  conditions,  we  have  from  (2.3) 

■<?{(’•,  (t, r(x))~^g~ 

(3-6) 

However,  we  can  solve  (3.6)  by  variation  of  constants  and  obtain 

'v(x')  f ^ 

- =  Mt> T(x)  •  J  dw*  •  g^(r,(t,r(x))DtAx)-  (3-7) 

The  result  can  be  verified  by  differentiation,  because  (3.6)  has  a  unique  solution.  Therefore, 
with  x  =  £o, *(*())> 


dMt,T(x) 

dx 


[  a~*  :  t  <  ^  (T 

=  dwr-g(r,Zt,r{x)) - fx - dr  +  J 


dV(t,x) 

dx 


dw*  ■  g^{r,^,r(x))DtAx) '  cUo,rOo)) 


+  cs(Zt,T(x))E>t,T(x)} 


=  E 


fT 

]  dw*  ■  g((r,  $OAX0))P()AX0)  ■  c(€o,T(xo)) 
+  ct(Z0,T(x0))D0,T(x0 )  I  Ft  Do,t(x 0)> 


and  the  result  follows. 

REMARK  3.2.  The  result  extends  immediately  to  functions  for  which  a  generalized  Ito 
formula  holds;  this  class  includes  convex  functions  and  differences  of  two  convex  functions. 
See  Karatzas  and  Shreve  [10]. 


4.  Hedging  Portfolios. 

It  is  shown  in  Harrison  and  Pliska  [9]  that  hedging  policies  arise  from  a  martingale 
representation  under  an  equivalent  measure.  Consequently,  we  give  an  application  of  The¬ 
orem  3.1  in  this  section. 

Consider  a  vector  of  d  stocks 


s=(sl,...,sdy 

whose  prices  are  described  by  a  system  of  stochastic  differential  equations  of  the  form 

d 

dS\  =  si(»%st)dt  +  £  \ij(t,St)dwl). 

3= 1 
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Models  of  this  kind  axe  usual  in  finance.  When  the  pl  and  \{j  are  constant  we  have  the 
familiar  log-normal  stock  price.  For  economic  reasons,  so  that  the  claim  is  attainable,  see 
[9],  the  number  of  sources  of  noise,  that  is  the  dimension  of  the  Brownian  motion  w,  is 
taken  equal  to  the  number  of  stocks.  A t  =  A(f,  S)  =  (A (j(t,  S ))  is,  therefore,  a  dx  d  matrix. 

We  suppose  A  is  non-singular,  three  times  differentiable  in  S,  and  that  A and  all 
derivatives  of  A  have  at  most  linear  growth  in  S.  Writing  p(t,  S )  =  (/x*(f,  5), . . . ,  p^(t,  S))1 
we  also  suppose  p  is  three  times  differentiable  in  S  with  all  derivatives  having  at  most  linear 
growth  in  S. 

We  suppose  the  stocks  pay  nc  dividends.  However,  suppose  there  is  a  bond  5^  with 
a  fixed  interest  rate  r,  so  S®  =  ert.  The  discounted  stock  price  vector  &  =  (£), . . .  ,^)/  is 
then  (t  e~rtSt  so 


Writing 


di\  =  ~r)dt  +  A ij{t,ertit)dw^j. 

j- 1 


&t  =  A(i,&)  = 


and  p  =  (r,  r, . . . ,  r )'  equation  (4.1)  can  be  written 

d£t  =  At((p  -  p)dt  -f  A tdw).  (4.2) 

As  in  Section  2,  there  is  a  flow  of  diffeomorphPms  x  — >  £^(r)  associated  with  this  system, 
and  their  non-singular  Jacobians  Dsj. 

In  the  terminology  of  Harrison  and  Pliska,  [9],  the  return  process  Yf  —  (Y^, Yf~) 
is  here  given  by 

dY  =  (p  —  p)dt  +  A  dw.  (4.3) 

The  drift  term  in  (4.3)  can  be  removed  by  applying  the  Girsanov  change  of  measure.  Write 
p(t,S)  =  A —  p)  and  define  the  martingale  M  by 


Then 


rt 

Mi  -  1  -  /  Msrj(s,  S)' dws. 

JO 

Mi  —  exp  (  —  rj'dwg  -  i  j  \Vs\2dsj 


is  the  Radon-Nikodym  derivative  of  a  probability  measure  P.  Furthermore,  under  P, 
wt=wt+  /  r)(s,  S)fds  is  a  standard  Brownian  motion.  Consequently,  under  P 

Jo 


dYt  =  A  tdwt 


and 


d&  =  /\th-tdwt-  (4.4) 

Therefore,  the  discounted  stock  price  process  £  is  a  martingale  under  P  so  P  is  a  ‘risk- 
neutral’  measure.  _  _  _  _ 

Consider  a  function  ip  :  R?  R,  where  ip  is  twice  differentiable  and  ip  and  ipx  are  of 
at  most  linear  growth  in  x.  For  some  future  time  T  >  t  we  shall  be  interested  in  findmg  the 
current  price  (i.e.,  current  valuation  at  time  t),  of  a  contingent  claim  of  the  form  ^(Sy). 
It  is  convenient  to  work  with  the  discounted  stock  price,  so  we  consider  equivalently  the 
current  value  of  _ 

:=  ^WTiT)' 

ip  has  linear  growth,  so  we  may  define  the  square  integrable  P  martingale  N  by 

Nt  =  £(</>(fr)  |  Ft],  0  <  t  <  T. 

As  in  Harrison  and  Pliska,  [9],  if  we  can  express  N  in  the  form 

Nt  =  EmT))  +  Wdu 

JO 

then  <f>  —  is  a  hedge  portfolio  that  generates  the  contingent  claim.  However, 

we  can  apply  Theorem  3.1  to  derive  immediately: 

Theorem  4.1.  ^ 

Nt  =  EmT)\  +  f 

Jo 

where 

-r  rT 

<f>{s)  =  E  /  ^(tt,  eru^o,u(a;o))^0,u(^0)^  ’  ^(£o,T(*o)) 

J$ 

+  ^ Uo,t(xo))do,t{xo )  I  Fs  D0,l(x o)- 

PROOF.  From  Theorem  3.1,  under  measure  P 

Nt  =  E[ip((T)\  +  f  jsdws 
Jo 

where 

~r  fT 

7s  -  E  /  ^D0iU{xo)dwu  •  ^(fo,T(*o)) 

Js 

+  1l’^0,T(xo))D0,T{xo)  I  FS  Do,l(x o)A(£o,s(zo))As- 

Because  d&  =  AjA tdwt,  <p(s)  has  the  stated  form. 

REMARKS  4.2.  Note  that  if  rj  is  not  a  function  of  £,  (which  is  certainly  the  situation 
in  the  usual  log-normal  case  where  /z  and  A  are  constant),  rj^  is  zero  and  the  first  term  in 
<p  vanishes. 

The  bond  component  <f>®  in  the  portfolio  is  given  by 

d 

$  =  o  <<<r 

*=1 

and  Nt  is  the  price  associated  with  the  contingent  claim  at  time  t. 
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5.  Examples. 

Stock  price  dynamics  for  which  the  hedging  policy  <j>  can  be  evaluated  in  closed  form 
appear  hard  to  find.  However,  if  we  consider  a  vector  of  log-normal  stock  prices  we  can 
re-derive  the  Black-Scholes  results.  Suppose,  therefore,  that  the  vector  of  stock  prices 
S  =  (S*, . . . ,  S^)1  evolves  according  to  the  equations 

d 

dS\  =  S\ (ji'dt  +  Y,  xijdwt)  (5.1) 

3  = 1 


where  /i  =  (fA, . . .  ,//*)  and  A  =  are  constant.  The  discounted  stock  price  £  is  then 
given  by  (4.2). 

Consider  a  contingent  claim  which  consists  of  d  European  call  options  with  expiry 
dates  T\  <  T<i  <  ■  •  •  <  Tj  and  exercise  prices  cj_, . . .  ,c^,  respectively.  Then 

d  d 

m.-<Td)=  £/'«(), T^o))  =  Z((&,TtM-W-rTt)+- 

k=  1  k=l 


The  4'^  are  convex  functions,  so  applying  the  generalized  Ito  differentiation  rule  of  [10] 
Theorem  3.1  is  valid  as  noted  in  Remark  3.2  with 

=  (0, . . . ,  0,  /{{fo,  >  cte-rT‘ },  0, ....  0). 

From  (5.1)  we  see  that  the  Jacobian  Dqj  is  just  the  diagonal  matrix 


and  its  inverse  is 


»0,t  = 


exp  { -  ( 53  Ai M  -  \  «n<) } 


exP  {  -  (  .E  xdj™t  "  \  addl)  } 
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(The  explicit,  exponential  form  of  the  solution  shows  Dqj  is  independent  of  xq.)  Thus, 
the  trading  strategy  <f>k  that  generates  the  contingent  claim  ^(£y  )  is 


>~1 

,s 


4(5/  =  E[4>faQiTk(xo))Do,Tk  I 

=  (o,. . .  ,0,E  I{£o ,Tfc  >  ckt~rTh}  exp  |  Y^j  Xkj(™JTk  ~ 


d 

E 

i=i 


~  \  akk(Tk  ~  5)}  I  ?s  ,  0,  •  •  • ,  o) , 


for  0  <  s  <  Tk.  Note  that  (f>k(s)  =  0  for  s  >  Tk,  i.e.,  <f>k(s)  stops  at  Tk.  However,  from 
(5-1), 

d 

4,Tk(x' o)  =  4  exP  {  E  XkAk  -  2  akkTk]  >  cke~rTk 

j- 1 


iff 


a  cl 

E  xkAk  >  los  (^f)  +  (o  akk  -  r)Tk  =  <*b  say; 

i= 1  xo 


(5.2) 


that  is,  iff 


d 

1 

j= 1 


E  Xkj(™Tk  ~  *%)  >  <*k  ~  E  Xkj™l 


d 

E 

3= 1 


Now,  under  P,  X)  Xkj(™Tk  ~  *s  norrnaMy  distributed  with  mean  zero,  variance 
j=l 

akk(Tk  —  s ),  find  is  independent  of  Therefore,  the  nonzero  component  of  </>j,(s)  is 

2 

“  ^  Ui 

5) 


/  <*  exp{x  -  ^  akk{Tk  -  s)}  exp  {  ■  -  -  }  ■  .=  ===== 

J<Xk-J2  Xkjw*  2  ''*akk(Tk  S))  V^nakk(1k~ 

~[ x  ~  akk(Tk  ~  s)\2 


j= 1 

roo 


—  I  d  . exp  <  • 

J<*\ fc-  E  ^ 


■>  ax 

^akk^k  ~  s)  y/^’7Takk('^'k  ~  s ) 


j=i 


/  -v2/2  dy 

~  /-*~EA-*f  *  -akk(Tk-s)  6  ^2^ 

V  akk(Tk-3) 


-*( 


+  E  Xkj™3s  +  gfcfcffi;  -  g) 

Vakk(Tk  ~s) 


)■ 
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Again  from  (1),  ]T)  A kjws  —  l°g  f  ~,S  £ — )  +  ^  akks’  which  together  with  (5.2)  gives 
3= 1  ^0 


3- 

( 


4(5) 


(  log  -  \  akk(Tk  -  s)  +  rTk 


\ 


Vakk(Tk  ~  s) 

or,  in  terms  of  the  (nondiscounted)  price  Sk, 

^  /'log  (It)  ~  d  ahk  ~  r)(Tk  ~SX 


V 


) 


) 


4(s)  = 


V 


V 


-  ~) 


V 


J 


(5.3) 


) 


o  <  S  <  Tk.  Therefore,  the  trading  strategy  (j>  generating  ?/>(T1; . . .  ,Tfc)  =  £  ) 

/fc=l 

can  be  written,  with  a  slight  abuse  of  notation,  as  <f>(s)  =  (<f>i(s), . . . ,  4>d(s))' ,  where 


/ 


h(*)  =  j{s<ta.}$ 


loS  (f^)  -  akk  -  r)(Tk  ~s)^ 


V 


Vakk(Tk~s) 


(5.4) 


J 


d  _ 


Finally,  we  calculate  the  price  of  the  claim  E[ip(Ti, . . .  ,Td)]  =  f)  E[ipk(£T  )]  similarly: 

k= 1 

E^*(fri)l  =  EMrv'r,,‘)+ 

fc=l  ik=l 


d~d  d 

~  X]  ^kj^Tk  >  afc}  (^0exP  {  X]  ^kj^Tk  ~  p  “  cke~rTk ) 

fc=l  J=1  ;=1 


-£** 
k= l 


loS  (Ij-)  +  ($  aJbifc+r)TJfc 


\ 


V 


aJfejfer* 


( loe  4-  frr  a.uu  4-  rlTV 


"  \  ck  ) 


\ 


V 


'■  Z  A,  ft  /  a  j - 


) 


(where  we  have  used  (ft  =  S$,  k  =  l,...,d).  When  d  =  1  the  above  result  reduces  to  the 
well-known  Black-Scholes  formula. 
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FILTERING  FOR  A  LOGISTIC  EQUATION 

R.  J.  Elliott 
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1.  INTRODUCTION 

Filtering  is  a  mathematical  theory  of  estimating  a  “signal”  from  noisy  observations.  It  has  had 
striking  successes  in  many  areas  of  engineering.  For  example,  the  linear  Kalman  filter  described 
in  Section  2  below  was  derived  in  1960  and  is  credited  with  a  large  role  in  the  U.S.  space  program. 
Once  the  theory  and  techniques  are  more  widely  known  it  is  likely  that  filtering  will  have  important 
applications  in  other  areas.  Applications  of  filtering  to  biological  problems  can  be  found  in  Refs 
[1-4],  for  example. 

In  Section  2  the  form  of  the  linear  Kalman  filter  is  derived.  The  analogous  equation  in  the 
nonlinear  situation  is  obtained  in  Section  3.  This  equation  has  a  quadratic  term,  so  in  Section  4 
the  Zakai  equation  for  the  unnormalized  density  is  established.  Finally,  in  Section  5,  a  logistic 
equation  with  some  noise  is  discussed.  Using  a  technique  of  Kunita  [5],  it  is  shown  how  the 
unnormalized  conditional  estimate  can  be  expressed  without  using  stochastic  integrals  with  respect 
to  the  observation  process.  Fuller  details  of  the  ideas  described  can  be  found  in  Refs  [6, 7]. 

2.  THE  KALMAN  FILTER 

The  basic  idea  of  the  model  is  that  there  is  a  signal  process  {x,},  which  cannot  be  observed 
directly,  and  a  related  observation  process  {y,}.  The  object  is  to  obtain  the  “best  estimate”  of  x,, 
given  the  history  of  y  up  to  time  t.  To  illustrate  the  ideas  we  first  discuss  the  linear  Kalman  filter. 
For  simplicity  the  processes  will  be  one-dimensional. 

Suppose  the  signal  process  is  given  by  the  following  linear  stochastic  differential  equation: 

dx,  =  Ax,  At  +  C  d  IF, , 

where  W,  is  a  Brownian  motion.  Then 

x,  =  x0  +  J  Ax, As  +  CIV,. 

The  observation  process  will  be  assumed  to  be  of  the  form 

Ay,  =  Hx,At  +  ABn 

where  B,  is  a  second  Brownian  motion  independent  of  W.  Assume  x0~  N(0,  P0)  and  yo  =  0. 

The  filtering  problem  is  to  calculate  a  recursive  expression  for  the  "best  estimate”  of  xn  given 
{.V -re¬ 
write  Y,  =  o{y, :  s  <  /}  for  the  a -field  generated  by  y,  up  to  time  t  and 

x,-E[x,\Y,). 

Then  x,  is  the  “best  estimate”  in  the  sense  that  it  minimizes  E(x,  -  z)2  over  all  Y, -measurable, 
square  integrable  random  variables  z. 

Calculating  x,  is  a  Hilbert  space  projection  problem.  Suppose  (Cl,  F,  P)  is  the  probability  space 
on  which  our  random  variables  are  defined.  Consider  L\(Cl),  the  space  of  square  integrable  random 
variables  of  zero  mean.  For  X,  Y  eLl(Cl),  the  inner  product  is  just  E(XY). 
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The  process  v  has  the  following  properties: 

(i)  v,  is  a  Y,  Brownian  motion,  i.e.  v,  is  a  T,-martingale  and  <v>,  =  /,  where  <v>  is  the 
unique  predictable  process  such  that  v2-  <v>  is  a  martingale. 

(ii)  All  Y, -martingales  are  stochastic  integrals  with  respect  to  vf,  i.e.  if  M,  is  a  T-martin- 
gale,  then  there  is  a  process  g,  which  is  Y, -predictable  and 


M,= 


8s  dv,. 


Suppose  the  signal  process  is  an  fv -semimartingale  £,  given  by 


Zt  =  Co  + 


cqds 


Here  a  is  an  F-adapted  process  such  that 


F  a2ds  <  co, 

Jo 

q0  is  an  F0-measurable  random  variable  with  E£,\  <  00,  and  tj,  is  a  square  integrable  F, -martingale. 
There  is  a  unique  predictable  process  <i/,  F>  such  that  i],B,  —  (q,  B},  is  a  martingale.  We  shall 
suppose  this  is  of  the  form 


psds. 


Theorem  1.  Write  q,  =  E[q,jY,]  for  the  filtered  estimate  of  £,,  given  Y,.  Then 


l  =  lo  + 


asds  + 


(£Mx,)-£Mx,)  +  fi,)  dv5. 


Proof.  Write 


Pi  =  l  “  <f0  “ 


a,ds. 


Then  it  is  easy  to  check  that  fi  is  a  Y, -martingale.  Consequently,  by  property  (ii)  above  there  is 
a  process  gs,  such  that 

Pt=  f 


Again  we  wish  to  determine  g.  To  do  this  recall 

q,  =  q0+  f  a,  ds+r/n 


!/■ 

4/  =  Co  +  J  + 


8s  dv,, 


K.-JVt 


)  dj  +  B, 


and 


T,  =  jo^Cjd^  +  v,- 

From  equations  (1)  and  (3),  using  the  Ito  rule: 


(1) 

(2) 

(3) 

(4) 


Z,y,=  q,[h(q,)ds  +dDs]  + 


y,(f*,ds  +  dijs)  + 


p,ds. 
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Conditioning  each  side  on  7,,  we  see  that 


E{Z,y,\Y,}  =  y,l  =  f  [&(£,)  +  *«,  +  &]  ds  +  N) , 

Jo 

where  N]  is  a  7, -martingale.  However,  from  equations  (2)  and  (4),  using  the  Ito  rule: 

C'  -  C'  C‘  C' 

y,l  =  iHZs)  is  +  l  dv,  +  y,(as  is  +  g,  dvj  +  g,  ds 
Jo  Jo  Jo  Jo 

=  f  [iAO+yA+g,]is  +  ni. 


where  N*  is  a  7, -martingale.  The  decompositions  of  y£,  in  equations  (5)  and  (6)  must  be  the  same, 
so 

N)  =  N* 


£,/»(&) + yA  +  A  •-  IkQ  +yA+gt. 

This  gives  gs  =  qsh(^s)  —  £/?(£,)  +  and  the  result  follows.  End  of  proof. 
Suppose  that  x,  is  the  solution  of  the  stochastic  differential  equation 


d.v,  =/(/,  x,)  dt  +  ff(x,)  dn>, . 


That  is, 


x,  —  Xq  +  f(s,  x,)  d.v  + 


’i 

ff(*,)dw,. 

Jo 


If  F  is  a  C2  function,  the  Ito  differential  rule  tells  us  that 


Fix,)  =  F(x0)  + 


F(x 0)  +  Fx(xs)f(s,  xs)  ds 

*/  j  Pi 

+  Ft(Y,)<;(xJ)dH'J  +  -  F„(xs)<r(x,)2ds. 

Jo  1  Jo 


Let  us  use  the  notation 


n,(F)  =  E[f(x,)\Y,\ 


Then  IT,  can  be  thought  of  as  the  conditional  distribution  of  x,,  given  7,,  so  that 


* 

W)  = 


F(x)  n,(d.v). 


Suppose  w  is  independent  of  the  observation  noise  B.  Applying  Theorem  1  to  the  semimartingale 
F(x,)  gives 

n,(F)  =  n0(F)  +  ['  ns(FF)  ds  +  f '  [n  ,m  -  d  v  (8) 


EF(x )  =f(s,  x)FJx)  +  \a{xfFxx{x). 

This  equation  gives  an  infinite  dimensional  recursive  equation  for  the  filtered  estimate  nXT7). 

If  we  consider  the  case  f(s,  x)  =  A,  <7(x,)  =  C,  F(x)  =  x  and  h(x)  =  Hz,  equation  (8)  reduces  to 
the  Kalman  filter  derived  in  Section  2. 
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4.  THE  UNNORMALIZED  FILTERING  EQUATION 

There  are  two  difficulties  with  equation  (8).  Firstly,  it  is  in  some  sense  quadratic  in  ri  and, 
secondly,  it  is  driven  by  the  innovations  process  v.  Suppose  we  are  considering  processes  defined 
on  [0,  T],  Define  a  new  probability  measure  P0  on  (Q,  F)  by 


dF0  r  r  l  CT  1 
^  =  eXPL  Jo  /,2('Vl)d"  [ 


If /t  is  of  linear  growth,  say  |/i(*)|  *ZK(1  +  |*|),  then  P0  is  a  probability  measure  and  under  P0,  y, 
is  actually  a  Brownian  motion.  This  is  a  result  of  Girsanov’s  theorem  [6,  Theorem  13  14]  Write 
E0  for  expectation  with  respect  to  P0  and 


4,-^AWd,-!  ' h\xs)ds  . 


Then,  using  a  Baye’s  type  theorem, 

n,(£)  =  £[£(*,  )|T,] 

_  £0[/l,F(.v,)|y,] 

EM.\Y,] 

°,{F) 

where  a,(F)  -  £0[/l,£(x,)j}',]  is  an  unnormalized  conditional  distribution  of  F{x,). 

We  first  obtain  a  semimartingale  expression  for  <r,(l).  Using  the  Ito  differentiation  rule: 


d/1,  =  fi(x,)A,  dy,. 


/!,=  !+  h(x,)A,dy, 


That  is, 


so  A,  is  a  (f„  £0)-martingale.  Consequently,  A,  =  £0[/l,|  TJ  is  a  V, -martingale  so  there  is  some 
Y, -predictable  process  y„  such  that 


rt 

l',=  l+  ; 

Jo 


To  determine  y,  consider,  using  equation  (9), 


V  f'  C' 

y,A,=  A,dy,+  ysh(x,)Asdys  +  Ash(x,)ds. 

Jo  Jo  Jo 


Conditioning  on  Y,  under  measure  P0,  we  have 


£o[^/l,|T,]  =  y,/l',=  (*  AJi(xs)ds  +  M), 


where  M)  is  a  (Y,,  £0)-martingale.  However,  using  equation  (10), 
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Again,  the  decompositions  (11)  and  (12)  must  be  the  same,  so 


M J  =  M] 


y,  =  Ash(xs)  =  ^[/l,/!^,)!^]. 


Using  Baye’s  rule  again,  we  see  that 


y,  =  A3ns(h(x,))- 


Therefore, 


A,=  l  +  J^rWv^djv  (13) 

(Note  denotes  conditioning  under  measure  P0,  while  n  denotes  conditioning  under  the  original 
measure  P.)  Now  equation  (13)  has  the  unique  solution 


A-t  —  exp  n,  {h )  dy,  -  H  II,  (/t  )2  ds 
Jo  zJo 


=  tr,(l). 

Recall  ff,(P)  =  ffAOnXF).  Forming  the  product  of  equations  (8)  and  (13)  therefore,  and  using  the 
Ito  rule,  we  have  the  following  result: 

Theorem  2.  a, (F)  satisfies  the  “ Zakai  equation”, 

<r,(F)  =  (70(F)  +  f  c r,(fF)  d*  +  f '  o,{hF)  dy„  (14) 


with  initial  condition 


a0(F)  =  U0(F)  =  E[F{x0)). 


This  equation  is  linear  in  o  and  is  diiven  by  the  observation  process  y.  There  is  a  one-to-one 
correspondence  between  solutions  of  the  Zakai  equation  and  solutions  of  the  nonlinear  filtering 
equation  (8):  whenever  <r,(F)  satisfies  equation  (14),  then  a,(F)/a,{  1)  satisfies  equation  (8),  and 
whenever  fI,(F)  satisfies  equation  (8), 


n,(F)exPr  f'nJ(/I)djJ-l  pn,(/o2ds 

-Jo  zJo 


satisfies  equation  (13). 


5.  EXAMPLE 


Suppose  the  state  of  the  system  under  investigation  is  described  by  the  following  logistic-type 
equation: 

dN,  =  ?.N,{\  -txNJdt+edw,. 

That  is,  N  satisfies  the  usual  logistic-type  equation  with  a  small  amount  of  noise  represented  by 
c  div. 

The  observation  process  will  be  of  the  form 

dy,  =  kN,dt  +  d  B,. 
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Again,  if  noise  were  not  present  in  the  observation  process  we  would  have 

N=k-<^1 
N'  *  dr’ 

so  that  N,  could  be  determined  from  the  rate  of  growth  of  the  observation  process.  Write 
Ul{N)  =  E[N,\Yl],  so  the  innovations  process  is 


v,  =  y,~ 


{N)ds. 


Suppose  the  observation  noise  B  and  state,  or  signal,  noise  it  are  independent.  Then  equation  (8) 
for  the  filtered  estimate  I"I,(iV)  gives 

n ,(N,)  =  n0(N0)  +  f '  n,(AAf,(i  -  *ns))  6s  +  k  P  (n ,(a2)  -  n ,(NS)2)  dv,. 


Equation  (13)  for  the  unnormalized  filtered  estimate  a,(N)  gives 

a,{N.)  =  n0(W0)  +  P <r,(AA(,(l  - aN,)) ds+k  [' <rs(N2s)dys. 
Jo  Jo 

Again  consider  a  C2  function  F  so  that,  using  the  Ito  rule, 

F(N,)  =  F(N0)+  P  AF(NS)  ds  +  c  [' FN(N,)dws. 

Jo  Jo 


AF(N,)  =  XN,( I  -  aNs)Fs(Ns)  +  -  Fnn(Ns). 


Therefore,  equation  (8)  gives 


n,(F)  =  n0(F)  +  f  n,.(^F) ds  +  k 

Jo  Jo 


If  fl,  is  given  by  a  conditional  density  p,  then 


l,(F)  = 

JR 


F(x)p{t,  x,y)dx. 


Consequently,  p  is  given  by  the  equation 


%t  rt 

p(kx,y)  =p(0,x,y)+  A*p{s,x,y)ds  +k  p(s,x,y)(x  -xs) dvs. 
Jo  Jo 


d  e2  d2d>(x) 

A  *</,(*)  =  -  -  [/x(l  -  ttxWx)]  +  - 

is  the  adjoint  of  A.  This  equation  is  not  linear  in  p  because 


/% 

xs=  xp(s,  x,_ 
Jr 


However,  if  we  consider  the  related  unnormalized  conditional  density  q,  given  by 

q(t,x)  =  A,p(t,x), 


F(x)q(t,x)ds. 
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Furthermore,  q  is  given  by  the  linear  equation 


q(t,  x )  =  /5(0,  .v,  j>)+  I  A  *q(s,  x)  ds  +  £  xq(s,  x)  d ys, 
Jo  Jo 

which  has  y  as  input. 

Finally,  let  us  consider  again  the  Zakai  equation  for  ff,(F): 


a,{F)  =  a0(F)  +  P  ffs(AF)  ds  +  k  f'  as(N,F(Ns))  dv,. 
Jo  Jo 


In  terms  of  Stratonovich  integrals  this  is 

=  <r0(F)  + 1'  <7 s(aF(Ns)  -  ^  N;F(nSJ  ds  +  k  °  a$(N,F(N,))  °  dys. 
Consider  the  following  operators  defined  on  functions  F(N): 

L(t)F(N)  =  AF(N)  -  -  N2F(N) 

ft,F(N)  =  F(N)cxp(Ny,) 
and 

/*,' 1  F(N)  =  F(N)cxp(  —  Ny,). 

Writing  A(/V)  for  XN(\  —  aN),  see  that 

r *  f) p'  a 2 1,2 

[MlL(t)n;']F(N)  =  lw  +  [&(Nl)-(2yl}~+  ~~  ~  A(N ,)y,  -  ~  N2  F(N), 

so  n,L(t)nk'  is  a  second-order  operator. 

If  we  consider  N,  as  the  solution  of  the  system 

N,  =  N0  +  \'[A(Ns)-c2ys]ds  +  cw, 


and  consider  the  expectation,  given  Yn  of 


wc  have,  writing 


C  r  t  r  2  2  k  ”1^ 

F(yV,)expj  A(N<)ys--N;  ds  >, 

y,(F)  =  £(V(iV,)expj  "-f  -  A(Ns)y,  -  ~  B]  ds  J  Y.'j, 

v,(F)  =  v0(F)  +  I '  v,(/t, L(s)n; 1  )F{NS)  ds. 

Jo 


If  we  now  calculate  v,(;r,F),  we  have 


v,(/t,  F)  =  v0(F)  +  vs(n,L(s)n;')(n,F)  ds  +k 


^vsU‘s(NsF(  N, 


That  is,  the  solution  of  the  Zakai  equation  is  given  by 


vA&F)  -  £^exp(yV,_y,)F(yV,)exp|  ^  -  A(N,)ys  -  ^  N]  dsj  K  j. 


i 
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The  advantage  of  this  expression  is  that  it  involves  no  stochastic  integrals.  The  observation 
trajectory  y  appears  just  as  a  parameter.  Also,  the  operator  n,L(t)p f-1  differs  from  L(t)  only  by 
terms  of  less  than  second  order.  For  details  of  the  method  in  this  section  see  Ref.  [5]. 
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Abstract 

Using  a  perturbation  of  the  rate  of  a  Poisson  process,  and  an  inverse  time  change,  an 
integration  by  parts  formula  is  obtained.  This  enables  a  new  form  of  the  integrand  in  a 
martingale  representation  result  to  be  obtained. 
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Integration  by  Parts  for  Poisson  Processes 


1.  Introduction. 

In  his  paper  (3]  on  the  Malliavin  Calculus  Bismut  obtains  an  integration  by  parts 
formi  ’a  for  a  diffusion  by  considering  a  small  perturbation  of  the  trajectories  and  then 
compensating  for  this  by  using  a  Girsanov  change  of  measure.  That  is,  suppose  £  denotes 
the  original  trajectory  and  ££  the  perturbation.  Let  E  (resp.  Ee)  denote  expectation  with 
respect  to  the  original  measure  (resp.  the  measure  after  the  Girsanov  transformation). 
Then  for  any  bounded,  differentiable  function  c,  it  is  the  case  that 

£[c(0]  =  (1-1) 

The  left  side  of  this  equation  is  independent  of  e  and  Bismut  obtains  his  integration  by 
parts  formula  by  differentiating  in  e  and  putting  e  =  0.  Integration  by  parts  formulae 
for  Markov  jump  processes  have  been  obtained  by  Bass  and  Cranston  [1],  and  Bichteler, 
Gravereaux  and  Jacod  [2].  Again,  the  variation  of  the  trajectories  considered  by  these 
authors  consists  of  perturbing  the  size  of  the  jumps. 

A  Poisson  process  is  a  counting  process,  and  all  jumps  are  of  unit  size.  Consequently, 
a  perturbation  of  the  trajectories  of  the  kind  considered  in  [1]  and  [2]  does  not  make 
sense.  Instead  we  consider  below  a  Girsanov  change  of  measure  which  alters  the  rate  of 
the  Poisson  process  by  a  small  amount.  This  is  then  compensated  by  considering  a  time 
change  of  the  process  under  the  new  measure.  An  identity  analogous  to  (1.1)  is  obtained 
and  the  integration  by  parts  formula  follows  by  differentiating  with  respect  to  a  parameter 
£  and  putting  e  =  0.  The  case  where  the  function  depends  only  on  finitely  many  jumps  is 
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discussed  first,  and  the  general  case,  for  a  functional  of  the  Poisson  process  over  the  time 
interval  [0, 1],  is  then  deduced. 

There  is  a  close  relation  between  integration  by  parts  formulae  and  martingale  repre¬ 
sentation  results.  It  is  well  known  that  any  uniformly  integrable  martingale  on  the  sigma 
fields  generated  by  a  Poisson  process  can  be  represented  as  a  stochastic  integral  with  re¬ 
spect  to  the  associated  martingale.  The  integrand  can  be  obtained  by  considering  one 
jump  at  a  time  (though  the  precise  form  given  in  equation  (2.6)  does  not  appear  to  be 
in  the  literature).  What  is  interesting  is  that  the  integration  by  parts  method  gives  an 
alternative  expression  for  this  integrand,  which  does  involve  a  derivative  of  the  functional 
of  the  process.  The  equality  of  these  two  expressions  is  verified  in  the  appendix  when 
the  functional  depends  on  finitely  many  jump  times.  This  expression  for  the  integrand  is 
similar  to  that  obtained  by  Clark  [5]  for  functionals  of  Brownian  motion. 

2.  Martingale  Representation  and  Time  Change. 

Let  N  be  a  Poisson  process  on  (Q,jF,(^Ft),.P)  with  jump  times  !Z\,...,Tn, _  We 

shall  write  To  =  0.  Let  G(Ti , . . . ,  T„, . . . )  be  an  integrable  function  of  T\ , . . . ,  T„,  — 
Consider  the  martingale  M  defined  by: 

Mt  :=  E[G{T\ , T„, ... )  |  Ft).  (2.1) 

For  n  >  1,  write 

r-\Tn)  =  MTn  -  MTn_x 

=  E[G\?Tn)-E\G\?rn_X  (2.2) 


2 


FYom  Theorem  T9,  Chapter  3  of  [4],  the  martingale  M  defined  by  (2.1)  has  the 


representation: 

Mt  =  E[G ]  -f  f  gadQa 
Jo 


(2.3) 


where  Qt  =  Nt  —  t, 


9s  =  9'(s) 


—  f'(s  —  Ti)  —  e*~Ti 


(2.4) 


on  {Tg  <  s  <  T,4-i  },  and 


f\s)  =  E[MTi+x  Ti+i  -  Ti  =  s]. 

Since  =  <r{Ti , . . .  ,T,},  T,+i  -  T;  is  independent  of  and  exponentially 

distributed,  so 


r°° 

E[I{Ti+l>s}MTi+l  I  Ft,)  =  /  ■f{u+r.>J}-®[^T,+1  T,-+i  - T{  =  u)e~udu 

Jo 

=  f\u)e~udu.  (2.5) 

9(*— Tj)vo 


From  (2.4)  and  (2.5),  we  have 


g\s)  =  E[G  |  Ti+1  =  s]  -  e^T'E[JTi+1>,(?  |  Jjj). 


(2.6) 


FYom  (2.5) 


Mt;  =  E[MT;+l  |  Tt,\ 

=  /  ft(u)e~udu. 

Jo 


(2.7) 
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Also,  tx  can  be  written  as 


£''(s)  =  /*'(s- TO-  Mo¬ 


using  (2.4),  (2.7)  and  (2.8),  we  have 


g\s)  =  e(s)  +  f  t(uy-udu. 
J)Ti,s) 


Throughout  the  rest  of  this  paper  we  let  {u{,  t  >  0}  be  a  real  predictable 
satisfying: 

(i)  {ut,  t  >  0}  is  positive  and  a.s.  bounded,  |it*|  <  B  a.s.  say. 

(ii)  There  exists  a  bounded  interval,  say,  [0,  6],  such  that  u3(w)  =  0  if  s  <£  [0,6],  a. 
For  e  >  0,  consider  the  martingale: 


ft  :=  /  eutdQt 
Jo 

=  ^  eugAN,  —  /  eugds. 

OC  Jo 


Define  the  family  of  exponentials 


a;  :=  exp(X,  -  \(X\X%)  n  (1  +  hX,)e-*x- 

0  <3<t 

=  Yl  (l+eu,ANa)exp(-  f  eu,ds). 

o  <3<t  x  Jo  ' 

Then  {Af,  t  >  0}  satisfies  the  equation: 


dX„ 


AJ  =  1+  f 
Jo 

. t 

=  1+1  A \_euadQ9 
Jo 

and  {Af,  t  >  0}  is  a  martingale.  (See  [6].) 


(2.8) 


(2.9) 

process 


(2.10) 


(2.11) 


(2.12) 
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LEMMA  2.1.  {A£,  t  >  0}  is  a  uniformly  integrable  martingale.  Hence  exists  and  a 
new  probability  measure  Pc  can  be  defined  by 


dP£ 

dP 


Proof.  It  suffices  to  show  that  the  martingale  {A*,  t  >  0}  is  square  integrable. 

Recall  u  vanishes  outside  the  interval  [0, 6]  and  |u*  j  <  B  a.s.  By  (2.12)  and  Ito’s  rule, 

(a;)2  =  i  +  2  /'  A;_aA‘,  +  T  (auu.an.)2 

J0  0<»<I 

=  1  +  2  ['  A‘_dk‘,  +  /'(A  ’.)W,dQ,+ 

Jo  Jo  Jo 

For  0  <  t  <  b, 

£[(A?)2]  =  1+  f  E[(Al_)W$)ds 
Jo 

<l  +  e2B2  f  E[{A\f)ds. 

Jo 

So  by  Gronwall’s  inequality, 

£[(A')2]  <  exp(e2B2<) 

<  exp  (e2B2b)  0  <  t  <  b. 

A\  is  contant  for  t  >  b.  Hence  the  martingale  {A',  t  >  0}  is  square  integrable. 

A^,  >  0  a.s.  and  FfA^,]  =  1  so  we  can  define  a  new  probability  measure  Pc  by  putting 


Then  the  process  {Q* }  defined  by 


Qct  :=  Nt  -  [  (1  +  eus)ds 
J  o 


is  an  ( J-t )  martingale  under  Pe  (see  [7]). 

Now  define 

<t>e(t):=  /  (1  +eut)ds 
Jo 

Let  ifre(t)  =  Then  t/>e(<^£(t))  =  t  so 


(2.13) 


Mt)  =  /  T 


1 


o  1  +  ev.^€(a) 


ds. 


(2.14) 


If  we  let  T{  =  T, /,,(<),  then  from  Theorem  T16,  Chapter  II  of  [4],  the  process  {TV/,  t  > 
0}  defined  by: 

Nte  :=  NMt)  (2.15) 


is  a  Poisson  process  on  (0, T,  (pf),Pe). 


3.  Integration  by  Parts. 

Suppose  G  is  a  function  of  the  first  n  jump  times  T\, . . . ,  Tn  of  a  Poisson  process  N. 
Since  4>t{t)  =  jfT,-  is  the  i-thjump  time  of  {Nt},  then  <^£(T,-)  is  the  z-th  jump  time  of 

the  process  {A^,(t)}.  Changing  the  rate  of  the  point  process  by  a  Girsanov  transformation, 
and  then  changing  the  time  scale  of  the  process,  we  have  the  following  result: 


THEOREM  3.1.  Let  G{T\,...,Tn)  be  bounded  with  bounded  first  partial  derivatives. 
Then 


E{(J\,dQ,)G(T„...,T„ )]  =  -£[£  J;G(T, . T„)jl  «.*].  (3.1) 
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r  ~~ 


Proof.  By  the  results  in  Section  2,  because  is  a  Poisson  process  under  Pe  with 

jump  times  <^(T,);  consequently 

E[G(TU . . .  ,T„)]  =  E‘{G(MT, MTn))) 

=  £[A'o0G(WT1),...,^(T„))]  (3.2) 

where  Ee[  ]  denotes  that  expectation  is  taken  with  respect  to  Pe.  Differentiating  (3.2) 
with  respect  to  e,  and  then  setting  e  =  0,  we  get: 

|  G(MTi) . «T„))|.„0] 

+  E[(je  A~),..0  GW.(Tl)....,«T„))|,.0]  =0.  (3.3) 

Prom  (2.12)  and  the  definition  of  A^, 


Noting  the  definition  of  <j>e,  (3.3)  becomes  (3.1)  and  the  proof  is  complete.  □ 

Remark  3-2.  Consider  a  function  H  of  the  form  H(Ti  A  1, . . . ,  Tn  A  1)  where  H  is 
bounded  and  has  bounded  first  derivatives.  Applying  Theorem  3.1  to  G(Ti, . . .  ,Tn)  = 
H(T\  A  1, . . . ,  Tn  A  1)  and  noting  that 

. t„)  =  Air(r,M . r.Ai)/jiS„ 

we  have  the  following: 

COROLLARY  3.3.  If  H (Ti  A  1, ...  ,T„  A  1)  is  bounded  and  has  bounded  first  derivatives, 
then 

E[[  j\,dQs')ff(T1Al,...,TnAl)] 


Remark  3.4.  Recall  the  martingale  representation  (2.3)  and  (2.4),  or 


rTn 

G(Tlt...tTn)  =  E[G}  +  /  g.dQ„ 

Jo 


where 


9  s  =  g'~l(s)  for  T,-!  <  s  <T,. 


If  we  substitute  (3.5)  into  the  left  hand  side  of  (3.1),  we  get 


E[{  Jo  u»dQa'j  (e[G)  +  I  gadQt^=E^J  utgsds]  =  E  J  (3.6) 


where  gt  =  0  for  s  >  Tn. 

Also,  if  we  consider  the  measure  pc  defined  by 


**(*)  =  E  4rG(T1,...yTn)ST;(dt). 

«=i  ati 

Then  the  right  hand  side  of  (3.1)  can  be  written 


-£[EJ:G<r' . «.* 

~~E[Jo  J0  U»dsKclt)\ 

=  ~e\^J  /i(s,oo)u,ds| 

i(T— -r"H' 


Lot  C»  =  X)  /r,>a^0(Ti, . . .  ,Tn).  Then  there  exists  a  predictable  projection  C* 


of  C,  such  that  for  each  s, 


Cl  =  £(C.  |  Jf,-]  «u>. 


Also  for  any  predictable  process  {«,,  s  >  0}, 


E[usCt]  =  E[utE[Ct  |  Ft-}\ 


=  E{utC;). 


Let  H  be  the  family  of  subsets  of  [0,  oo)  x  f2  of  the  form  {0}  x  F0  and  (s,  t ]  x  F,  where 
Fo  €  F0  and  F  6  F»  for  s  <  t.  Recall  that  the  predictable  cr-field  is  generated  by  H. 
Taking  u  =  /{o}xF0  or  u  —  /(4|<]xF)  then  u  satisfies  the  hypothesis  in  Section  2,  so  (3.6), 
(3.7)  and  (3.8)  hold  for  these  u.  Also  because  of  (3.8),  on  comparing  (3.6)  and  (3.7),  we 


E  [  J°°  u»9,ds\  =~E[Jo  usC;ds 


holds  for  all  u  which  are  indicators  of  sets  in  H.  Since  H  generates  the  predictable  cr-field 
and  the  processes  g  and  C*  are  predictable,  therefore  we  have  proved  the  following  result: 


Proposition  3.5. 


Now  suppose  H(T\  A  1, . . .  ,Tn  A  1)  is  bounded  with  bounded  first  partial  derivatives. 


FYom  Section  2,  H  has  the  representation: 


y*Tn  Al 

H(T1Al,...,TnAl)  =  E[H}+  /  g,dQ, 

Jo 

where  ga  =  g'~l(s)  for  T;_i  A  1  <  3  <  Ti  A  1. 

An  argument  similar  to  the  above  shows  that 
r  n  dH 

»,  =  A 1 . r» A  n  i 


(3.10) 


(3.11) 
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The  form  of  g  given  in  Section  2  and  that  given  in  (3.9)  are  at  first  sight  rather  different. 
A  direct  proof  of  their  equality  is  sketched  in  the  Appendix.  Next  we  have  the  following 
integration  by  parts  formula: 


Hn(Tx , . . . ,  Tn)  =  E{G(Ti  A  1, . . . ,  T„  A  1,  (T„  +  Sn+1 )  A  1, . . . , 

(T„  +  Sn+1  d - b  Sn+i)  A  1, . . .  )  j  J~Tn] 

=  ES[G{T\  A  1, . . .  ,T„  A  1,  ( Tn  +  ■S'n+i)  A  1, ... , 

(Tn  +  S, i+i -i - b  Sn+{)  A  1, . . .)],  (3.13) 
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where  S*  =2*  -  Tk-i  for  k  >  1,  and  the  last  expectation  Es  in  (3.13)  is  taken  only  over 


the  random  variables  Fn+1 , . . . ,  Sn+i, . . . ,  and  the  Tx , . . . ,  Tn  are  given.  From  (3.1), 

rT< 


\  i  bUr  r  f)un  rTi 

UgdQsj  Hn{T\ , . . . ,  T„)j  =-gjE;[^-(T1,...,Tn)|  u,ds 

fn  r Tn 

— (Ti,...,T„)  /  i£ads 

It  Jo 


r<9tfn 


(3.14) 


And  from  (3.13), 
dHn 


.  fi  ^ 

-(r,,...,rn)  =  Bs[^-G(T,  A1,...,T„M,(T„  +  S„+1)M,...)] 

OO  Q 

=  £S[Z)^:G(ri  A1,(T„  +  5n+i)  a  1, . . . )/r,<i 

i=n  1 
°o  £ 

=  A  1, . . . ,Tn  A  1,(T„  +  5„+1)  A  1,. . .  |  ^rn 


Hence 


E 


\dHn ,  fTn 

— (T,,...,Tn)^  «.& 


=  i? 


I  u»ds  53 -E ["^"CA  A  1,. . .  ,Tn  A  1, .. .  )/jj<i  I 


— ►  0  as  n  — ►  oo 

by  the  hypotheses  on  {u,}  and  G. 
Also  for  1  <  i  <  n  —  1, 

■8Hn 


(3.15) 


E 


r  ru 

— (Ti,...,Tn)  /  ut 


*■  Jo 

r  a 


ds 


—  A  A  1,...)It(<i  |  Ft„  J  Usds 

=  A  1,...,T„  A  1,...)  J  '  uadslTi<i 


(3.16) 
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ri-r 


Letting  n  — ■»  oo  in  (3.14),  because  of  (3.15)  and  (3.16),  we  obtain  (3.12).  □ 

We  conclude  with  the  following  theorem: 

THEOREM  3.7.  Suppose  M  is  the  right  continuous  martingale 

Mt  =  E[G(T\  A  1, . . .  Tn  A  1, . . . )  |  Ft). 


Then 


Mt 


= m  +  f 

Jo 


Q»dQt , 


where 


r  90 

s.  =  -E  [  J2 M,  •  •  •  ,T„  A  1, . . . )  | 

Proof.  The  proof  is  similar  to  the  one  for  Proposition  3.5. 


a.s. 


(3.17) 


4.  Appendix. 

We  now  give  a  direct  proof  that  the  integrands  g  obtained  in  sections  2  and  3  are 
equal. 

The  idea  is  to  first  establish  the  equivalence  on  {t  <  Tj}  and  then  to  use  this  to 
establish  the  equivalence  on  {T,_i  <  t  <  T,-}  for  i  >  1  without  any  more  calculation.  First 
we  need  a  preparatory  lemma.  Given  a  bounded  measurable  function  R(x i, . . . ,  xn)  define 

^(l/tV ••»!/»;  Ii> * •  •  =  H(xi,...  y,-  +Xj_i,...,i/n  +  ij_i). 


Notice  for  later  use  that 


dyj 


7?(‘) 

rdR/dyj 


for  j  >  i. 


(4.1) 


Finally,  for  i  fixed,  let  Tj  =  Tj  —  T,-_i  for  j  >  i  and  notice  that  (T,-, . . .  ,Tn)  has  the  same 
probability  distribution  as  (7\,. . .  ,Tn_,-+1). 
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LEMMA  4.1.  If  (T\, . . .  ,  Tn)  are  the  successive  jump  times  of  a  Poisson  process,  then 
(*) 


E[R(Tu...,Tn)\fTi_l}  =  E{F£\Ti,...,Tn]  )j|(>i .  ^ . ^ 


(b) 


E{R(T\,. . .  ,Tn)  |  Ti,...,T,_i,  Ti=t] 


=  E[Fjl\Ti,. . .  ,  Tn;  |  T{  =  t  —  x,-_i] 


(xi,...,r{_i)ss(T’i,...)Tf_i) 


(c) 


JE[H(T, . t„)  |  r,)\. 


(Ti-i<«<T,} 


=  1, 


*ll  •  •  ‘  Jxi-l)  \  Ti  >  <  — Xj_l]| 


(*i . ii_i)=(Ti . T;_  O' 


The  proof  of  this  lemma  is  elementary  and  relies  mainly  on  the  independence  of 
(Ti, , T„)  from  T\ , . . . , T,_i ,  and  the  equality  ft  {T,_i  <  f  <  T,}  =  fl  {T,_i  <  t  < 
T1,};  see  [4],  Chapter  3,  Section  1. 

Now  we  are  ready  for  the  proof.  Note  first  that  T,  —  Fa~ ,  since  the  sigma  algebras 
are  complete. 

Step  1  We  show  equivalence  on  { T\  >  t}.  A  direct  calculation  and  an  integration 
by  parts  shows  that 


l{Tl>0(£?[G(T1,...,Tn)|T1=t]-etJB[l{ri>t}G(T1)...,Tn)] 

=  . r»)|r.  >i  - 
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By  part  (c)  of  the  lemma  and  by  using  the  facts  that  l{Tt>t}  is  measurable  and  that 
>«}  =  1  { Ti > t }  almost  sure  for  j  >  1,  we  find  that  this  last  expression  equals 

This  completes  Step  1. 

Step  2.  We  prove  the  equivalence  on  {T,_i  <  t  <  T,}.  First  use  (a)  and  (b)  of  the 
Lemma  to  show  that 

. Tn)  I T. . Ti- ,,  T,  =  (]  -  e'-T‘-JS[l(T,>,)G  |  *•„-,]) 

-  Ijr,., \  f(  =  t~  x,_ij 
-e‘  T-"E{F'a){fi,...,fr,-,  . . T; _ i) ' 

(4.2) 

Now  apply  Step  1  at  each  (xx,. .  .  ,x,_ j)  with  the  random  vector  (Ti, . . . ,  T„)  replaced  by 
(Ti, T„),  and  G(Ti , . . . ,  T„)  replaced  by  Fq\T{, . . . ,  T„;  —  ,  x,-_i )  and  x,-_i ) 

fixed  but  arbitrary.  Using  (4.1),  we  find  in  this  way  that  the  expression  in  (4.2)  is  equal  to 


n 

. 


;=« 


(*i,...,zi_i)=(Ti . Ti-,) 


But  from  part  (c)  of  the  Lemma,  this  equals 


j=i  3 


,  .n ,  dG 

=  1{Ti_i<<<ri}-E[X]1{ri_,<t<T;-}  ^(Ti,...,Tn)  j  7 , 

j=i  3 


n 

nfV'1 


=  1  1  <<<r; }  -S  [  1  {Ti- 1  <t<Ti }  3  {T>><}  ^(T!,...,Tn)  I  7t\ 

=  i)35-lSKU)-B[i)ima.)  fjrCr, . t.)  I  * 
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The  last  three  equalities  are  all  elementary.  This  completes  the  proof. 
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Abstract.  Using  a  differentiation  result  of  Blagovescenskii  and  Freidlin 
calculations  of  Bensoussan  are  simplified  and  the  adjoint  process  identified 
in  a  stochastic  control  problem  in  which  the  control  enters  both  the  drift 
and  diffusion  coefficients.  A  martingale  representation  result  of  Elliott  and 
Kohlmann  is  then  used  to  obtain  the  integrand  in  a  stochastic  integral,  and 
explicit  forward  and  backward  equations  satisfied  by  the  adjoint  process  are 
derived. 


I.  Introduction 

The  adjoint  process  in  stochastic  control  problems  has  been  investigated  in  several 
papers.  For  example,  see  the  works  of  Bismut  [4],  [5],  Davis  and  Varaiya  [7], 
Haussmann,  [13]-[15],  Kushncr  [16],  and  the  previous  papers  by  Baras  et  al.  [1], 
and  Elliott.and  Kohlman  [1 1],  [12].  In  most  of  these  papers  the  control  variable 
enters  only  the  drift  term.  However,  in  an  interesting  paper  [2],  Bensoussan 
considers  the  case  where  the  control  is  also  present  in  the  diffusion  coefficient.  By 
obtaining  the  Gateaux  derivative  of  the  cost  function  the  equation  satisfied  by  the 
adjoint  process  is  derived.  However,  a  martingale  representation  result  is  used  and 
this  equation  involves  an  unknown  integrand.  The  contributions  of  this  paper  are 
that,  by  using  results  of  Blagovescenskii  and  Freidlin  [3]  on  the  differentiability  of 
solutions  of  stochastic  differential  equations  that  depend  on  a  parameter,  the 


*  This  research  was  partially  supported  by  NSERC  under  Grant  A79M,  the  U.S.  Air  force  Office 
of  Scientific  Research  under  Contract  AFOSR-86-0332,  and  the  U.S.  Army  Research  Office  under 
Contract  DAAL03-87-K-0102. 
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calculations  of  Bensoussan  can  be  simplified,  and,  by  applying  the  expressions 
obtained  by  Elliott  and  Kohlmann  in  [9]  and  [10]  for  the  integrand  in  a  stochastic 
integral,  explicit  forward  and  backward  equations  for  the  adjoint  process  are 
obtained  when  the  optimal  control  is  Markov.  In  particular,  the  backward 
equation  is  a  nonstochastic  system  of  parabolic  partial  differential  equations  of  a 
novel  form,  though  it  is  probably  related  to  the  equation  obtained  in  the  recent 
paper  by  Davis  and  Spathopoulos  [6], 


2.  Stochastic  Dynamics 

Assume  the  state  of  the  system  is  described  by  the  following  equation: 

dx,  =  f(t,  x„  u)  dt  +  g(t,  x„  u)  d\v„  x,  e  RJ,  0  <  t  <  T.  (2.1) 

The  control  variable  u  will  take  values  in  a  compact,  convex  subset  U  of  some 
Euclidean  space  Rk.  We  assume: 

Al.  x0  e  Rd  is  given. 

A2.  /:  [0,  T]  x  Rd  x  U  -»  Rd  is  continuous,  and  continuously  differentiable  with 
respect  to  x,  u. 

A3,  g:  [0,  T]  x  Rd  x  U  -*  Rd  ®  R"  is  a  continuous  matrix-valued  function,  which 
is  ontinuously  differentiable  with  respect  to  x,  u.  The  columns  of  g  are  denoted  by 
gik)  for  k  =  1, . . . ,  n. 


A4.  There  is  a  constant  K  such  that 

(1  +  MV  ‘I fit,  x,  «)|  +  \fx(t.  x,  «)|  +  | fu(t,  x,  m)|  <  K, 

\g(t,  x,  m)|  +  \gx  (t,  x,  m)|  +  I gu(t,x,  i<)|  <  K. 

A5.  w  =  (w1, . . . ,  vt'")  is  an  n-dimensiona!  Brownian  motion  on  a  probability 
space  (Q,  F,  P\ 

The  right  continuous,  complete  filtration  generated  by  w  is  denoted  by  {F,}, 
0  <t  <:  T. 

Notation  2.1.  Write  L|[0,  7]  =  {v  =  v(:,  w)  e  L\Q  x  [0,  T];  dP  x  dt;  Rk ):  such 
that,  for  a.e.  t,  v(t,  •)  e  L2  (f2,  F,;  P,  Rk)}. 

Definition  2.2.  The  set  of  admissible  controls  is  the  set 
U  =  {a  s  Lf[0,  Tj:  v(t)  e  U  a.e.  a.s.}. 

Then  V  is  a  closed,  covex  subset  of  Lj[ 0,  T]. 


The  Optima!  Control  of  Diffusions 


231 


Remark  2.3.  For  each  u  e  U  there  is,  therefore,  a  unique  strong  solution  of  (2.1). 
We  write  x"  ,(x)  if  x  e  Rd,  0  <  s  <;  t  <  T,  for  the  solution  trajectory  given  by 

*“.,(*)  =  *  +  |  /(r,  x“  ,(-x),  ur )  dr  +  j  g(r,  x“  ,(x),  «r)  d»vr.  (2.2) 

Because  u,(w)  depends  only  on  t  and  w,  the  result  of  Blagovenscenskii  and 
Freidlin  [3]  extends  to  this  situation,  so  the  Jacobian  d.x“_  ,(x)/dx  =  Z)“ ,  exists  and 
is  the  solution  of 


=  I  + 


fx(ri  *“,r(*)>  ut)D“,r  dr  +  £  ^(r,  x“r(x),  «r)D“rr/w‘.  (2.3) 

S  fc  =  1  Js 


Here  7  is  the  d  x  d  identity  matrix.  In  fact,  if  the  coefficients  f  and  q  are  Ck  the  map 
x->x“,(x)is  Ck~'. 

Consider  the  matrix  process  H  defined  by 


HI,  =  /  - 


H“r(  fx(r,  x“  r(x),  uf)  -  £  gf(r,  x“  ,(*),  ur)2 )  dr 


~  £  Hlrg[k)(r,  xlr(x),  u,)dwi. 

k~  l  Js 


(2.4) 


Using  the  Ito  rule  we  see  r/(77“,7)“ ,)  =  0,  and  HUS_SDUS  S  -l  so 

=  W1- 

Write  || x“(x0) || ,  =  sup0siS(|Xo.,(Xo)l-  Then,  as  in  Lemma  2.1  of  [15],  for  any 
p,  1  ^  p  <  oo,  using  Gronwall’s  and  Jensen’s  inequalities 


Ix“(x0)rr  <C(  1  +  |x0r 


9(r,  xu0,r(x0),  ur)  dwf 


a.s.  for  some  constant  C.  Therefore,  using  Burkholder’s  inequality  and  hypothesis 


A4, 


||x"(xo)llr  is  in  Lp  for  1  £  p  <  oo. 

Write 

Ilf" Hr  =  sup  |D|U, 

0S3ST 

V  li  Hu  II  r  =  sup  |/7U0J. 

0  SssT 

■\  Then,  because  jx  and  gx  are  bounded,  an  application  of  Gronwall’s,  Jensen’s,  and 
Burkholder’s  inequalities  again  implies 


||  Du  ||  r  and  ))HU\\T  are  in  V, 


\  <  p  <  co. 


Cost  2.4.  We  assume  there  is  a  cost  associated  with  the  process,  made  up  of  a 
terminal  cost  and  a  running  cost, 


c(*o.t(*o))  + 


f  h(r,xu0<Xx0),u.)dr. 


Jo 

We  assume: 

A6.  |c(x)|  *•  | cx\  +  |cxx(x)|  ^  K(l  +  |x|<)  for  some  q  <  oo. 

A7.  h:  [0,  T]  x  Rd  x  U  -*  R  is  Borel  measurable  and  continuously  differentiable 
in  (x,  u). 


Furthermore 


IMf,  *>  «)l  ^  CiO  +  M)> 

|/i„(t,  X,  «) |  <C2(l  +  |x|). 


The  expected  cost  if  a  control  u  s  U  is  used  is,  therefore, 


J(u)  =  £ 


c(^0.  r  (*o))  + 


h(r,  x£  ,(x0),  ur)  dr 


We  assume  there  is  an  optimal  control  u*  e  U,  so  that 


J(u*)  <;  J(u) 
for  all  other  u  e  U. 


Notation  2.5.  We  write  x*  for  x“*,  D$  ,  for  /)£,,  etc. 


3.  Differentiability 

Assume  u*  e  U  is  an  optimal  control.  Consider  any  other  control  Del).  Then  for 

0e  [0,1] 

Ud(t)  =  U*(t)  +  0(v(t)-U*(t))6V. 

Now 

J(ue)>J(u*).  (3.1) 

If  the  Gateaux  derivative  J'(u*)  of  J ,  as  a  functional  on  the  Hilbert  space  L;;[0,  T], 
is  well  defined,  differentiating  (3.1)  in  6  implies 


(J'(u*).  t:(t)  -  >!*(!)}  >  0 
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Lemma  3.1.  Assume  neU  is  such  that  u$  =  u*  +  Qv  6  U  for  Oe  [0,  a].  Write 
<,(x0)  for  ^ie  trajectory  associated  with  uf.  Then  z,  =  dx00  t(x0)/d6\0=0  exists  a.s. 
and  is  the  unique  solution  of  the  equation 

*t 

z,  =  (fx(r,  x$.r(x0)>  u*)zr  +fu(r,  x$>r(x0),  u*)vr)  dr 
Jo 

+  £  (ffx(r,  x$_  ,(x0),  u*)zr  +  gf(r,  x$.  ,(x0),  u*)vr)  dw'.  (3.2) 

»=  1  Jo 


o.,(*o)  =  -'Co  +  [  f(r,  *o.r(*o)>  f*+  0Vr)  dr 
Jo 

+  £  9{i)(r,  Xo.r(x0),  u*  +  0vr)  dw‘r 

1=1  Jo 


and  the  result  follows  from  the  theorem  of  Blagovescenskii  and  Freidlin  [3]  on  the 
differentiability  of  solutions  of  stochastic  partial  differential  equations  which 
depend  on  a  parameter.  □ 

Comparing  (3.2)  for  z  and  (2.3)  for  D$ ,  =  dx*Jdx,  we  have  the  following 
result  by  variation  of  constants: 

Lemma  3.2.  Write 

no.,=  (£>o .r)~lf»(r)vrdr+  £  (£>£,)"  lg{f(r)vr  dw‘r 

Jo  1=1  Jo 

-  £  ['(D*(yrlgf(r)g%)vrdr.  (3.3) 

«=  1  Jo 

Then  z,  =  D^,q0  l. 

Proof  By  differentiating,  we  see  the  product  Dgy/0 ,i  satisfies  (3.2).  □ 


Lemma  3.3. 


dJ(uo) 

dO 


~~  —  *T  cx(xo,  t  (*o))^o,  r'/o.  r 

0  =  0  L 

+  (lifr,  xlr(x0),  u*)D^rq0ir  + hfr,xlr(x0),  u*)vr)dr  . 


JK)  =  ^(4.  r  Oo))  +  h(r,  x°0_  r(x0),  uj  (r))  drj. 
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The  result  of  Lemma  3.1  and  [3]  justifies  the  differentiation  in  0  giving 

dJ(u*)  f  rr 

=  E  cx{xlT(x0))zr  +  (hx(r ,  xf„  uf)zr  +  hv(r,  4>r,  u*)vr)  dr  . 

du  Q~o  [  Jo 

Substituting  z,  =  Do,,//0,,  (3-4)  follows.  □ 

Notation  3.4.  Consider  the  right  continuous  version  of  the  square  integrable 
martingale 

M,:  =  E  cx(xl  T  (x0))D%  T  +  [  hx(r,  x^X^o),  tf)#*.,  dr\F,  . 

Jo 

It  is  known  (see,  for  example,  [8])  that  M,  has  a  representation  as  a  stochastic 
integral 


M,~  e\cx(x$'T(x0))DIt  +  f  hx(r)D*r  dr  +  £  fy [dw\,  (3.5; 

L  JO  J  i=l  Js 

where  the  y‘  are  predictable  processes  such  that 

hT  (yf)2  dr  <  oo. 

L  Jo 

We  determine  the  y‘  below,  but  first  we  introduce  some  more  notation.  Write 


=  J  hx(r)Dl,dr. 

Definition  3.5.  The  adjoint  variable  is  the  process  defined  by 
Ps  =  UD  lsyl. 


Theorem  3.6. 


dJ(ut)  | 
dO 


=  £[  PJv(s)vs  ds  -  £  f  ps(/x\s)g{‘\s)vsds  +  hv(s)vsds 
9=0  LJ0  i=l  Jo  Jo 

+  Z  y‘,(Dts) “  lgvKs)vs  ds  .  (3 

i  =  1  J  0 


Proof.  First  note  that 


Cr  1o.  t 


=  T  Z,(Dl,rlfv(s)vsds 
0 

+  Z  UDWl9(Xd<  ~  Z  [  UDo. 

i=  1  Jo  1=1  JO 

n  rr  -  CT 

+  Z  y‘dh.s  dw[  -  hx(s)Dl5>i0,5ds 

i=  1  Jo  JO 

+  Z  I  y'fDfX1  gf(s)v,ds.  (3.7) 

i=l  Jo 
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Also 

dj(u*)  r  ct  fT 

— 7n~la=o  =  £  CT>/o.r+  +  hv(s)vsds  .  (3.8) 

dv  L  Jo  Jo 

substituting  (3.7)  in  (3.8)  and  using  the  definition  of  ps,  the  result  follows.  □ 


1.  Martingale  Representation 

Under  certain  conditions  the  minimum  cost  attainable  under  the  stochastic  open- 
oop  controls  is  equal  to  the  minimum  cost  attainable  under  Markov  feedback 
.'ontrols  of  the  form  u(s,  do.s(xo))-  See,  for  example,  [4]  and  [13].  If  «M  is  a  Markov 
;ontrol,  with  a  corresponding,  possibly  weak,  solution  trajectoij  c“M.  then  mm  can 
oe  considered  as  a  stochastic  open-loop  control  »M  ( a> )  by  setting 

uM(o>)  =  mm(s,  Zoms(x0,  <o)). 

This  means  the  control  in  effect  “follows”  its  original  trajectory  <“M  rather  than  any 
new  trajectory.  The  control  «M  is,  therefore,  similar  to  the  adjoint  strategies 
ntrodueed  by  Krylov.  The  point  of  considering  the  open-loop  control  «M  is  that 
when  we  consider  variations  in  the  state  trajectory  and  derivatives  of  the  map 
x  ->  cs((x),  the  control  does  not  react,  and  so  we  do  not  introduce  derivatives  in  the 
u  variable. 

We  assume  in  this  section  that  the  optimal  stochastic  open-loop  control  u*  is 
Markov.  We  can  then  determine  the  integrands  y‘  in  (3.5). 

Lemma  4.1.  For  1  <  i  <  n 

yi  =  (^s-9‘<  + PsffPwjvo.s- 


Proof.  Consider  a  stochastic  system  with  components  Xo.,(x0),  E>*_„  and  To,i> 
where 


fo..=  f  dr. 

Jo 


r  r  i 

P,Do.,  +  V0.,  =  M,  =  E  cx(xQ' t(x0))D*' t  +  hx(r)Dlrdr\F,  . 

Jo 


Writing  x  =  x$,(x0),  D  =  D%  ,,y  -  y0-(  by  the  Markov  property  this  expectation  is 
the  same  as 


E  cx(x*T(x  ))D,  tD  +  f  hx  ( r)D*,drD  +  y|x,  v,  #1  =  V(l ,  x.  v.  D). 

L  J<  J 

From  the  left-hand  side  this  also  equals  p,(x)D  +  y.  The  martingale  representation 
result  of  Elliott  and  Kohlmann  [9],  [10]  follows  from  the  Ito  formula,  using  the 
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differentiability  of  the  solutions  of  stochastic  differential  equations  as  functions  of 
their  initial  conditions,  and  equating  the  bounded  variation  terms  to  zero. 
Therefore, 


M,  =  E\ 


crD  + 


hxD  dr 


+  I 


i- J 


y'r  dw‘r 


=  1/(0,  -Y0, 0,I)+t  f  PAr)c/\r)Dtr  dw'r  +  f  f  p(r)cj?(r)Dt,  dwf 
i ~  i  Jo  t  = i  Jo 


That  is,  y‘r  =  (px(r)g0,(r)  +  p(r)g(J>(r))D$tr. 
Remarks  4.2.  Substituting  in  (3.6)  we  have 


(4.1) 

□ 


dJ( uf) 
dO 


=  E\ 


9  =  0 


PJxls)vJs+  X  Px(s)9{i)(s)9vXs)vsds 


Jo 


1=  1  Jo 


(s)vs  els 


Returning  to  the  perturbation 
«o(0  “  u*(0  +  0(v(t)  -  u*(t)) 
of  the  optimal  control,  we  have 


dJ(M,) 
dO ' 


>0. 


9  =  0 


That  is, 
r  rr 


/•: 


PtfJMv,  ~  «,*)  ds  +  £  [  px(s)g,,)(s)gtJ>(s)(i’s  -  u*)  ds 

O  1  =  1  Jo 

Ms)(6,  -  «*)  >  o 


(4.2) 


for  all  o  e  U.  Define  the  Hamiltonian  by 

n 

H(x,  V,  t,  p(0)  =  p(t)f(t,  X,  v)  +  £  px(t)gM(t ,  x,u?)gM(t,  x,  v)  +  h(f,  x,  v). 

t  =  J 

Then,  because  (4.2)  is  true  for  all  v  e  U,  we  have  the  following  result: 

Theorem  4.3.  If  u*  is  the  optimal  control,  then  a.e.  t  and  a.s.  w 
SH. 


(4.3) 


dv 


(x*.,(x  o)’  :‘f.  ?(0)-(v,  -  <)  >  o 


for  all  v  e  V. 
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Finally  we  derive  the  equations  satisfied  by  p.  We  first  show  n  satisfies 
forward  stochastic  partial  differential  equation.  P 


Theorem  4.4. 


P<  E  c.x(-vo,  r(x«))Dl  T  +  |  hx(r,  .vj,r(.v0),  u*)D*  r  dr 

~  J0  Prfx('r’  X*oM  u?)  dr  -  f  hx{r,  x*  ,(x0),  u*)  dr 

J  0 

+  )  Px(r)g(r,  xlAxJ,  u*)  dw, 

V  0 

n  ft 

~ I  r ■  4.,«,  »,*)  dr. 

Proof.  From  (4.1) 
w«  *  P-Dl,  +  To., 

~  E  Cx(x0.  T(xo))^0.  T  +  llx(r)D*  r  dr\ 

*-  Jo  *  J 

n  ft 

i=l  Vo 

Multiplying  b,  HI,  =  (X). „hosc  equalion  fc  given  by  ^  we  ^ 

ft  ■ 

~  P’  '  { *?•'<*>'  “-*)  -  i  9SV,  .xs„w,  n?)!)  * 

J oPr0x(r>xl'(x),u*)dw>r-  f  hx{r,xlr{x),u*)dr 

Jo 

+  t  f  />,('Vi,(r) dw'r  +  £  (' Prdfir) dw‘ 

,  =  ,Jo  f=l  Jo 

n  ft 

j  /*W'W(r)dr  -  X  I  p,gf(r)gf(r)dr 

1=1  Vo 

“  Po  ~  Jo  />r/*W  <fr-  J  hx(r)  dr 

+  \  Px(r)g(r)dwr-  l  f  px(r)g%)g%)dr. 

i « ! JO 
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Here 


p0  =  E  cx(xl  T(x0))Dl  r  +  |o  K(r,  *o,r(*o)> «  *)£<>.  r 

and  the  functions  fjj),  hx(r),  g(r),  g{‘}(r)  are  evaluated  at  the  argument 
(r,  *0.r(*0)>  «,*)• 


□ 


Remarks  4.5.  Although  (4.4)  is  a  forward  stochastic  partial  differential  equation 
for  p  it  does  not  appear  to  have  a  unique  solution;  certainly  giving  any  (constant) 
initial  condition  does  not  determine  a  corresponding  vector  function  solution  p. 
We  now  show  p  is  given  by  a  backward,  nonstochastic,  parabolic  system  of  partial 
differential  equations  which,  under  the  given  conditions,  has  a  unique  solution. 


Theorem  4.6.  p  is  the  solution  of  the  backward  parabolic  system  of  partial 
differential  equations 

(jr)  n 

Jt  +  Px(t)f(t)  +  hx(t)  -r  p(t)fx(l)  +  i  £  Pxx(t)g{i)(t)  ®  9{i)(t)  =  0 
with  terminal  condition 


Pr  =  cx(x\ 


Proof.  Consider  again  the  function 
M,  =  V(t,  x,  y,  D)  =  p,{x)D  +  y 
of  Lemma  4.1.  Writing  again  the  Ito  formula  we  have 

r dV  dV  dV 

V(t,  x,  y,  D )  =  K(0,  *0, 0,  /)  +  J  -a-  +  ^  •  f(r)  +  Yy  ■  h^Dlr 
dV  n  d2V 

+  Y)  '  fx(r)°0.r  +  2  X  Yf  dr 

+  the  two  stochastic  integral  terms  in  (4.1).  (4.5) 


However,  M,  =  K(r,  x,  y,  D)  is  a  martingale  so,  as  observed  in  the  proof  of  Lemma 
4.1,  the  dr  integral  (that  is,  the  bounded  variation  term)  must  be  identically  zero. 
Therefore,  the  dr  integrand  in  (4.5)  must  be  identically  zero,  so 


dV  dV  dV  dV  "  d2V 

Jr+Fx  f(r)  +  Jy '  H*(r)D*-r  +  dD '  f*(r)D*'r  +  2  t  eff-cjU)(r)  ®  2<,>(r)  =  °- 


Recall  V  =  p\x)D  +  y,  so 


p7  +  PxiOfU)  +  KiO  +  p(.t)fx(t)  +  i  X  Pxx 
dt  i=1 


(t)g(i,(t)®g,i,(t)jD$'t  =  0. 


J 

3 
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is  nonsingular,  so  we  see  that  p  is  the  solution  of 

do  n 

Y,  +  Px(0f(f)  +  hx(i  +  p(t)U 0  +  i  £  pxx(t)gw(t)  ®  g(iKt)  -  0 

Ot  ia  1 

with  terminal  condition 

Pr  =  cx(x).  □ 


5.  Conclusion 


Using  the  differentiability  result  of  [3]  the  proof  of  Bensoussan  [2]  is  simplified  and 
the  adjoint  process,  when  the  control  appears  in  the  diffusion  term,  is  obtained. 
Furthermore,  by  applying  the  martingale  representation  result  of  Elliott  and 
Kohlmann  [9],  [10],  explicit  equations  for  the  adjoint  process  are  established. 

Under  certain  conditions  (see  [5]  and  [15]),  if  the  optimal  control  is  Markov 
we  can  write 


V(s,  x )  =  E 
=  ES. 


c(Xo,  T  (*0»  +  h(r>  *o.r(*o)>  «,*)  dr\ F, 

Jt 


c(x*T(x))  + 


h(r,  x*,(x),  u*)  dr 


where  .x  =  .x$.,(x0)>  f°r  the  optimum  remaining  cost.  Then,  at  least  formally,  we  see 
ps(x)  is  the  gradient  K,(s,  ,x);  so  px  is  K«(s,  x). 
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MARTINGALES  ASSOCIATED  WITH  FINITE  MARKOV  CHAINS 

by 

ROBERT  J.  ELLIOTT 

1.  Introduction. 

In  a  recent  paper,  [1  j ,  Phillipe  Biane  introduced  martingales  Mk  associated 
with  the  different  jump  sizes  of  a  time  homogeneous,  finite  Markov  chain  and 
developed  homogeneous  chaos  expansions.  It  has  long  been  known  that  the  Kol¬ 
mogorov  equation  for  the  probability  densities  of  a  Markov  chain  gives  rise  to  a 
canonical  martingale  M.  The  modest  contributions  of  this  note,  are  that  working 
with  a  non-homogeneous  chain,  we  relate  Biane’s  martingales  Mk  to  M,  calculate 
the  quadratic  variation  of  M  and  thereby  that  of  the  Mk .  In  addition,  square  field 
identities  are  obtained  for  each  jump  size. 

For  0  <  i  <  N  write  e,  —  (0, 0, . . . ,  1, . . . ,  0)*  for  the  z-th  unit  (column)  vector 
*n  **"  >  (so  e0  =  (1)0,  ...,0)*  etc.).  Consider  the  (non-homogeneous)  Markov 
process  {A^},  t  >  0,  defined  on  a  probability  space  (£l,F,P),  whose  state  space, 
without  loss  of  generality,  can  be  identified  with  the  set  S  =  {eo^,..  .  ,e/y}. 
Write  Pi  =  P(Xf  =  ej),  0  <  i  <  N.  We  shall  suppose  that  for  some  family  of 
matrices  Af,  pt  ~  (p® , . . .  ,p^ )*  satisfies  the  forward  Kolmogorov  equation 

dpt 

Ht^AtPt'  <L1) 


At  —  ( aij{t ))  is,  therefore,  the  family  of  Q-matrices  of  the  process. 

It  has  long  been  known  (see,  for  example,  Liptser  and  Shiryayev  [4],  Elliott  [2]) 
that  the  process 


=  Xt  —  X o  —  /  ArXr-dr 
0 


/ 

JO 


(1.2) 


is  a  martingale.  (See  Lemma  2.3  below.) 
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Solving  (1.2)  by  ‘variation  of  constants’  we  can  immediately  write 

Xt  =  $(tt0)(xQ  +  J*$(0,rr1dMr)  (1.3) 

where  $  is  the  fundamental  matrix  of  the  generator  A.  Equation  (1.3)  is  a  mar¬ 
tingale  representation  result  which  in  turn  gives  a  representation  result  in  terms 
of  the  M *.  (By  iterating  this  representation  Biane’s  homogeneous  chaos  expan¬ 
sion  can  be  obtained;  this  is  quite  explicit,  in  terms  of  matrices  <3>  and  matrices 
associated  with  A.)  Functions  of  the  chain  are  just  given  by  vectors  in  and 

in  Section  4  ‘square  field’  identities  are  obtained  for  each  jump  ‘size’. 

2.  Markov  Chains. 

Consider  a  Markov  chain  {X^},  t  >  0.  with  state  space  S  =  {eg,  •  •  - , 
and  Q- matrix  generators  At-  We  shall  make  the  following  assumptions. 
Assumptions  2.1.  (i)  For  all  0  <  i,j  <  N  and  t  >  0 

k-,(0l  <  B'  (2.1) 

for  some  bound  B'\  write  B  =  B'  +  1. 

(ii)  For  all  0  <  i,j  <  N  and  t  >  0,  a{j(t)  >  0  if  i  ^  j  and,  (because  At  is  a 
(Q-matrix), 

««(*)  =  (2-2) 
j^i 

The  fundamental  transition  matrix  associated  with  A  will  be  denoted  by 
<£(t,  s),  so  with  I  the  (N  +  1)  x  (N  +  1)  identity  matrix, 

=  Hs,s)  =  I  (2.3) 


and 


=  (t,s)As,  =  (2.4) 

(If  At  is  constant  $(t,  s)  =  exp  A(t  —  s ).) 

BOUNDS  2.2.  For  a  matrix  C  =  (c,-?-)  consider  a  norm  \C\  =  max  |ctr?|.  Then 
for  all  i ,  \At\  <  B.  The  columns  of  <3?  are  probability  distributions  so  |$(t,  s)|  <  1 
for  all  t,  s. 

Consider  the  process  in  state  x  6  S  at  time  s  and  write  X3j(x )  for  its  state 
at  time  t  >  s.  i 


Then  E[Xsj{x)\  =  ESix[Xt\  =  Write  Ff  for  the  right  continuous  4 

complete  filtration  generated  by  a{Xr  :  s  <r  <t}  and  F®  =  Ft . 


ii  JiP® 


Lemma  2.3.  The  process  Mt=Xt-Xo~  f  ArXr-dr  is  an  {Ft}  martingale. 

JQ 

Proof.  Suppose  0  <s  <  t.  Then 

E{Mt  -  Ms  I  F»|  =  E[xt  -  Xs  -  j‘ ArXrdr  I  Fa  j 

~  E  —  X3  —  J  ArXrdr  j  Xs  j 

=  Es,Xt[Xt}  -Xs-  f  ArESfxs{Xr}dr 

J  s 

~  s )-%s  -Xs-  J  Ar$(r,  s)Xsdr  =  0  by  (2.3). 

Therefore, 

Xt=X°+ Jo  ArXrdr  +  Mt  =  *0  +  J  ArXr^dr  +  Mt 
where  M  is  an  (P)}  martingale. 

Notation  2.4.  If  *  =  (x0,xh...,xN)*  e  RN +1  then  diag  *  is  the  matrix 


Lemma  2.5. 


>/,  M)t  =  diag  ArXr-dr  -  diag  X r.)A}dr  -  [at{ diag  Xr-)d, 

Proof.  Recall  Xt  6  S  is  one  of  the  unit  vectors  e,\  Therefore, 


Xt®Xt  =  diag  Xf. 


Now  by  the- differentiation  rule 


rt 

Xt®Xt=X0®XQ+  /  Xr_  ®  (ArXr-)dr 

J  0 


d"  Xr _  (QdMr  +  J  (Aj*Xr_) 


®  Xr-dr 


+  J  dMr  ®  XT _  +  (M,  M)t  +  Nt 


where  Nt  is  the  Ft  martingale 


[M,M]t  -  (M,M)t. 


However,  a  simple  calculation  shows 


Xr-  ®  ( ArXr _)  =  (diag  Xr-)A* 


(ArXr-)  <g>  Xr-  =  Ar(diag  Xr-). 


Therefore, 


ft 

Xt  ®  Xt  ^  Xq  ®  Xq  +  j  (diag  X, —  )A*dr 

JO 

rt 

+  /  Ar(diag  Xr-)dr  +  (M,M)t  +  martingale.  (2.6) 

JO 

Also,  from  (2.5) 

Xt  <g)  Xt  -  diag  Xt  =  diag  Xq  +  diag  f  ArXr-dr  +  diag  M*.  (2.7) 

J  0 

The  semimartingale  decompositions  (2.6)  and  (2.7)  must  be  the  same,  so  equating 
the  predictable  terms 

(M,M)t  =  diag  J  ArXr-dr  —  J  (diag  XrS)A*dr  —  Ar(diag  Xr~)dr. 
We  next  note  the  following  representation  result: 


Lemma  2.6. 


Xt  =  $(i,0)(*0  +  J*${r,Q)-ldMr). 


Proof.  This  result  follows  immediately  by  ‘variation  of  constants  . 


Remarks 


o  n  a  c. — ~P  V.  c.  C  ppn  Ko  ronrpcpnf.pH  bv  a  vector 

.  1  •  XX  lUUUbXUU  VI  JLj  N.  V  V***  ~  - - - - - "rf 

m=(fo(t),---jN(t))*  zrN+1 


so  that  f(t,Xt)  =  f{tfXt  =  if(t),Xt)  where  (  ,  )  denotes  the  inner  product 
in  RN+l. 

We,  therefore,  have  the  following  differentiation  rule  and  representation  result: 


LEMMA  2.8.  Suppose  the  components  of  f(t)  are  differentiable  in  t.  Then 


nt,Xt)  =  f(0,X„)+  f \f'(r),Xr)dr  +  f \f(r),ArXr-)dr  +  [\f(r),dMr). 

Jo  Jo  Jo 

(2.9) 

Here,  /  (f(r),dMr)  is  an  i^-martingale.  Also, 

Jo 

f(t,Xt )  =  (/(*),*(<,  0)X0)  +  f  (f(t),${t>r)dMr).  (2.10) 

Jo 

This  gives  the  martingale  representation  of  f(t,X{). 

Remark  2.9.  With  an  obvious  abuse  of  notation,  if  the  jump  times  of  the 
chain  are  2\(u;),T2(tt>), . . . ,  we  can  write  down  a  ‘random  measure’  decomposition 
of  Xt  from  (1.2)  as 

Xt=X0+  l  J2(ei  -  Xr-)(Y1  ^Tk(v})(dr)^tk(w)(})  ~  aiXr-dr ) 

J0  i  k 

rt 

+  JQ  J2(ei  ~  Xr-)aiX r-dr’ 

because  —  Xr_)a,-^r_  =  Ar~Xr _.  Here,  f>Tk(w)(dr)  is  the  unit  mass  at 
2).(u>)  and,  with  XTk(w)  =  eik(u>)>  fyk(w)(i)  is  1  if  i  =  i'K(w )  and  0  otherwise. 
That  is, 

Mt=  f  £(«;  “  -  aiXr-dr)- 

J0  i  k 

This  representation  would  provide  another  means  of  calculating  ( M,M)t . 


3.  Shift  Operators. 

The  formulae  of  Section  2,  particularly  the  martingale  representations  (2.8) 
and  (2.10),  provide  basic  information  about  the  Markov  process  X.  However,  if  the 
‘size’  of  the  jumps  is  considered  some  other  expressions,  including  a  homogeneous 
chaos  expansion,  were  obtained  recently  by  Biane  [1].  We  wish  to  indicate  how  the 
results  of  Biane  relate  to  the  above  expressions.  First  we  introduce  some  notation. 

Notation  3.1.  Write  i  ©  j  for  addition  mod  ( N  +  1).  For  X$  €  S  = 
{e0iel,...,ei\r}>  say  %s  =  and  &  =  write 

Xs  =  ei@k- 


That  is,  X3  —*  X3  corresponds  to  a  cyclic  jump  of  size  k  in  the  index  of  the  unit 
vector  corresponding  to  the  state. 

Suppose  Xs-  =  e{  and  Xk_  =  ej,  where  j  =  i@k ,  then  clearly 

=  «,■;(*).  (3.1) 


We  now  wish  to  introduce  some  subsidiary  matrices  associated  with  A.s  = 
(ajj(s)).  These  can  best  be  explained  by  first  considering  the  3x3  case.  Suppose 


G00  a01  a02  \ 
A  —  (  aio  an  ai2 
■a20  a21  a21 


Then 


A1  := 


f~a10 

0 

G02 

a10 

“a21 

0 

\  0 

a21 

“a02 

(~a  20 

a01 

0 

° 

-a0i 

a12 

'  a20 

0 

— a12 

A^  := 


Note  that  if  A  is  a  Q-matrix  agi  +  a\{  +  a2i  =  0,  so  A1  +  A2  =  A. 

In  general,  if  As  =  (a,-j(s))  is  an  (N  +  1)  x  (JV  +  1)  Q-matrix,  Ak  is  obtained 
by  forming  a  matrix  from  the  fc-th  subdiagonal  (continued  as  a  superdiagonal), 
with  the  negative  of  the  column  entries  on  the  diagonal  and  zeros  elsewhere.  By 
construction,  A*  is  a  Q-matrix,  and  it  is  clearly  related  to  those  jumps  of  ‘size’  k. 
As  above, 

N 

^  =  E  a°- 


(3.2) 


k=  1 


Also, 


((X*.rA,Xs-)(X*_  -  X ■,_)  =  4x,_. 


(3.3) 


so 


N 


Y,  {(Xs-YAsX3-)(Xks_  -  X3_)  =  A5X5_. 
k=l 


(3.4) 


/VI  t 

We  also  wish  to  introduce  matrices  A  ,  k  ^  0,  whose  off-diagonal  entries 
are  the  (positive)  square  roots  of  those  of  Ak ,  and  whose  diagonal  entries  are  the 


negative  of  that  square  root  in  the  same  column.  That  is,  in  the  (3  x  3)  case  above: 


For  k  — 


1,. 


^  — x/^lO 

0 

V^02  \ 

A1  := 

V^Io 

-V^2l 

0 

^  0 

\/G2l 

-V°02' 

/~v/«20 

Vm 

0  ^ 

A*:= 

0 

-v^oT 

v/°12  • 

0 

-Jm! 

N  write 

4  =  ((X*_)*A 

s*s-)~ 

l,2(xk_y, 

so  is  a  predictable  process. 

DEFINITION  3.2.  In  our  notation  the  matrices  introduced  by  Biane  [1] 
are,  for  k  =  1, . . . ,  N 


Mt=  E  ((Xka-fAaXaJ)-ll‘lI(Xa=Xka.)-l\(Xka.),AaXa.)ll'ldS. 


0  <3<t 


(3.5) 


Lemma  3.3.  Fork  = 


A3  •  dMs. 


Proof.  First  note 

(4-)*  ■  dX,  =  (x*L)*  •  AX,  =  (xk_ )*  ■  (Xs  -  Xa _)  =  r(Xa  =  Xk_).  (3.6) 


and  the  result  follows  from  (3.6). 


Lemma  3.4.  Fork  =  \,...,N,  {Mk,Mk)t  —  t. 

Proof.  M[;  =  j\k-  dMs,  so 

(Mk,Mk)t  =  [  Akd{M,M)3(Aky 
JO 

=  j‘((xk_)'Aaxs-y1/2(xk_y 

•  (diag  (j45X5_)  -  (diag  XS-)A*S  -  43(diag  X5_)) 


Now  for  k  ^  0: 


■(Xky((Xk.)'A,X,-)-1/2is. 


(X*_)*di«g  Xs_  =  0  =  (diag  X,-)(Xk_) 


(Xk_y  ■  (diag  (ASX,-))  ■  (X*_)  =  (X*_)M,X,_. 

Therefore,  (Mk,Mk)t  =  /  ds  =  t.  □ 

Jo 

Remarks  3.5.  For  k  ^  £,  Mk  and  have  no  common  jumps,  so  [Mk, 

0  and  =  0.  Therefore,  M1, . . . ,  are  a  family  of  orthogonal 

martingales,  each  of  which  has  predictable  variation  t. 

Having  expressed  Mk  in  terms  of  M  we  now  wish  to  express  M  in  terms  of 
the  Mk. 

THEOREM  3.6.  M*  =  £  f  AkXs-dMk,  so  the  Mk  form  a  basis. 

k= 1  JO 

Proof.  From  (3.6)  first  note  that 


k  \  _  /  vk  n* 


ix,  =  £(**_  -  ■  Ms- 

k=l 


Therefore, 


Xt-XQ=  f  A9Xs-d$  +  Mi  =  f dXs 

Jo  Jo 

An. 


=  E  /  (*«*-  -  **-)(*»-)*  '  (A,X,-i>  +  dMs).  (3.8)  * 

r.  i  J  0 


ii!i'  nwwMi  i 


By  definition  dMk  =  ((Xk_)*AsXs-)-lf2(Xk_)*  •  dMs  so  (**_)*  .  dMs  = 
((Xk_)*AsXs-)^^2dMk.  Substituting  in  (3.8) 

ft  N 

xt-x 0  = yo  ^(.^_-x3_)((4_rAsx3_)ds 

i — 1 
N  ft 

+  E  i,  -  X,-){(X*_)*ASX3-)11  iM$- 


Prom  (3.3)  and  (3.4)  this  equals 


rt  N  ft 

=  /  Asxs-ds  +  y  /  AksXs~dM f. 

Jo  ri  Jo 


Comparing  (3.7)  and  (3.9)  we  see 

iV 

M<  =  E  /  %*•-<&}■ 

i  *  JO 


(3.10) 


4.  Discrete  Derivatives  for  Different  Jump  Sizes. 

Consider  a  function  /  on  S  =  {et-}.  For  simplicity  we  suppose  /  is  constant  in 
time.  Then,  as  noted  in  Section  2,  /  is  represented  by  a  vector  /  =  (/o,  •  •  •  >  )* 

and 

f(Xt)  =  (/,*<)  =  (/,*o)  +  t(f,ArXr-)dr  +  {/,  Mt) 

JO 

rt  N  r‘ 

=  (f,X0}  +  /  (f,ArXr-)dr+£  /  (f,AkrXr.)dMk. 

JO  £=i  JO  (4.1) 


from  (3.9),  and  this  is 


rt  N  y 

=  (f,X o)+  /  (A*rf,Xr-)dr+Y,  /  {(Ak)'f,Xr-)iMk.  (4.2) 

JO  Jo 

We  new  re-establish  the  ‘square  field’  formula  of  Biane  (lj  by  calculating 
f(Xt )2  in  two  ways. 

Lemma  4.1.  A*f2  -  2/  •  A*f  =  £  ((4)*/)2. 

k=l 

Proof.  Function  multiplication  is  pointwise  in  each  coordinate,  so  /2  corre¬ 
sponds  to  the  vector  (/g , . . . ,  /jy)*,  and 

.1  N  ft 

S2(Xt)  =  (f2,X 0)+  /  (^,/2,Zr_)«ir+^  /  «,Akryf2,Xr-)dMk 

JO  £1  *  (4.3) 

=  (/(^i)j2- 


Using  the  differentiation  ruli  this  also  equals 


=  f(X0)2  +  2  f  f(XT_)df(Xr)  +  lf(X),  f(X)]t 

Jo 

=  f(Xa?+  2  f  (f,Xr-)(Atf,Xr-)dr 

Jo 

N  ft 

+  2  Y,  /  +  \fm,f(X)],. 

r  i  vQ 


[f(X)J(X))t=  £  &f(Xr)Af(Xr) 
0  <r« 


=  E  E  <Ur'r/,^r-)2(AMrfc)2 

ib=10<r<f 
iV  t 

=  £  f  ((AkTff,Xr-)2((X^rATXT-)-ll2dMkT 

k=lJ0 

N 

+  E  /  ((4)*f,Xr-)2dr,  from  (3.5). 
fc=r° 


Substituting  in  (4.4) 


/(Jf,)2  =  /{AT0)2  +  2  f  (/, Xr-)(A‘f, X r.)dr 

Jo 


N  rl 

+  2E  L  {f,Xr-)((Akr)*f,Xr-)dMl 


"  rt 


+  E  f  ((AkTrf,xr.)2[(xk_rArxr.)-ll2dMk 

i  •'O 


/V  rt 

.  X  ^  I  //  j&\*  c  tr  l  2  * 

+  2^  /  / > ^r—)  ar. 

k=iJ0 


The  special  semimartingales  (4.3)  and  (4.5)  are  equal,  so  equating  the  bounded. 


variation  terms 


(A*rf2,Xr..)  =  2 [f,Xr-)(£l,Xr-)  +  £  ((I*)7,*r->! 

Jb=l 


That  is,  as  functions  on  S 


£  ((AkrYff  =  a;/2  -  2/  .  A?/. 

£=1 

□ 

(A^)*  corresponds  to  a  discrete  derivative  of  ‘amount’,  or  in  ‘direction’  k. 
However,  the  algebra  suggests  that  (Ak )2  should  be  related  to 
A  more  specific  relation  is  now  obtained. 

Lemma  4.2.  For  k  =  1,  — ,  iV 

((#r/)2  =  (A‘)*/2-2/-(A*)7- 
Proof.  From  the  form  of  aA'  and  A^,  for  any  /  6 

(Ak)*f  =  (ako(-fo  +  /fc)>aJfe©l,l(”/l  +  /jb©l)> 

•  •  •  *  a*©yv,A(-/iv  +  /jfe©w))i 

(Ak)*f2  =  (flfco(-/o  +  /fe)»afe©l,l("/l  +  /fc©l)» 

•  •  •  >  ak®N,N(~fN  +  /?©jv))» 

iAk)*f  =  (V5fco(-/o  +  /fc)>  v/aJfc©i,i(-/l  +  /fc©l)» 

•  •  • » >/ ak®N,N(-fN  +  fk®N ))• 

Therefore,  as  function  multiplication  is  pointwise,  that  is  coordinatewise: 

((Afc)*/)2  =  (ajfeo(/o  ~2fofk+  fk)*”-iak®N,N(fN  ~2fNfkeN  +  /?©#)) 

f((Ak)*f)  =  (cjto(-/o  +  /o/it)>--->afc©iV,A(“/A  +fNfk®N))- 

Operating  coordinatewise,  for  example, 

(~fj  +  /fc©j)  “  2(“/j  +  /j/fc©j)  =  fj  "  2fjfk®j  +  /jfc©j 

and  the  result  follows.  □ 

Finally,  we  note  that  substituting  (3.10)  in  (2.9)  we  have 


Now  Xr—  is  a.s.  equal  to  XT  which  equals 


Xr  =  «(r,0)(^0+  £  /  *(r2,0 )-1AilXT1-iM%). 
k2  =  1  J0 

Substituting  in  (5.1)  we  have 

N  ft 

Xt=$(t,0)X0+Y,  /  mr)A^(r,0)X0dM^ 

jb=r° 

N  .t  rx 

+  E  E  /  /  W>n)^<Kri*r2)^Xri-dM%dM%. 

kx=lk2=l  J0  1/0 

Iterating  this  process  we  obtain  the  homogeneous  chaos  expansions  of  Biane  [1], 
(see  also  Elliott  and  Kohlmann  [3]),  in  terms  of  the  non-homogeneous  transition 
matrices  $  and  the  matrices  A 
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A  Partially  Observed  Control  Problem  for  Markov  Chains 


Robert  J.  Elliott 


1.  Introduction. 

In  this  paper  a  finite  state  space,  continuous  time  Markov  chain  is  considered.  The 
state  space  of  the  chain  is  taken  to  be,  without  loss  of  generality,  the  set  of  unit  vectors 
5  =  {e,},  e;  =  (0,0,...,1,...,0)*  of  RN+1 ,  thus  facilitating  the  use  of  linear  algebra. 
Basic  martingales  associated  with  the  Markov  chain  are  identified  and  the  solution  to  a 
natural  filtering  problem  associated  with  the  chain  is  given.  This  describes  the  recursive 
estimation  of  the  state  of  the  Markov  chain  if  only  the  total  number  of  jumps  is  observed. 
Such  a  filtering  formula  can  be  obtained  by  specializing  results  in  the  book  of  B  remaud  [5]; 
however,  using  the  basic  martingales  our  notation  and  framework  is  quioe  simple.  The  re¬ 
lated  Zakai  equation  for  the  unnormaiized  conditional  distribution  is  then  obtained.  In 
Section  5  the  optimal  control  of  a  Markov  chain  is  discussed  in  the  partially  observed  case, 
when  only  the  toted  number  of  jumps  is  known  to  the  controller.  This  control  problem  is 
treated  using  the  ‘separation  principle’,  by  discussing  the  control  of  the  unnormalized  dis¬ 
tribution  given  by  the  Zakai  equation  —  that  is,  the  filtering  problem  has  been  separated 
from  the  control  problem.  The  Zakai  equation  is  a  linear,  vector  equation  driven  by  a  stan¬ 
dard  Poisson  process.  Because  the  observation  process,  which  counts  the  number  of  jumps, 
is  correlated  with  the  state  process  the  signal  and  observation  processes  are  correlated,  so, 
in  contrast  to  earlier  work  on  controlled  Markov  chains,  the  control  variable  occurs  in  the 


‘diffusion’  coefficient  of  the  Zakai  equation,  multiplying  the  compensated  Poisson  process 
noise.  A  minimum  principle  is  obtained  by  adapting  techniques  of  Bensoussan  [1]  and 
calculating  a  Gateaux  derivative.  Finally,  if  the  optimal  control  is  Markov  the  integrand 
in  a  martingale  representation  can  be  obtained  more  explicitly.  This  enables  the  adjoint 
process  to  be  described,  and  new  forward  and  backward  equations  satisfied  by  the  adjoint 
process  are  derived. 


In  addition  to  the  book  of  B  remaud  [5],  other  works  which  discuss  the  control  and 
filtering  of  jump  processes  include  [3],  [4],  [6],  [12]  and  [13]. 


2.  Markov  Chain. 

For  0  <  i  <  N  write  e:-  =  (0, . . . ,  1, . . . ,  0)*  for  the  i-th  unit  (column)  vector  in  RN+l . 
Consider  the  Markov  process  {X<},  t  >  0,  defined  on  a  probability  space  (ft,  F,  P),  whose 
state  space  is  the  set 

S  =  {eo,ei,...,ew}. 

Write  p\  =  P(Xt  =  ei ),  0  <  i  <  N.  We  shall  suppose  that  for  some  family  of 
matrices  A*,  pt  =  (p?, . . .  pfO*  satisfies  the  forward  Kolmogorov  equation 

%-**■ 

At  =  (aij(t)),  t  >  0,  is,  therefore,  the  family  of  Q  matrices  of  the  process.  We  shall 
suppose  |a,j(t)|  <  B  for  all  i,j  and  t  >  0.  Because  At  is  a  Q-matrix 

o«w=-E^<w-  (2-2> 

The  fundamental  transition  matrix  associated  with  A  will  be  denoted  by  <3?(f,  s),  so 
with  I  the  (N  +  1)  x  (N  +  1)  identity  matrix 

=  »(«,»)-/  (2.3) 

= -Ht,»)A„  *(«,*)  =  I.  (2.4) 

(If  At  is  constant  $(t,  s)  =  exp (t  —  s)A) 

Consider  the  process  in  state  x  G  S  at  time  s  and  write  Xtit(x )  for  its  state  at  the  later 
time  t  >  s.  Then  P[XS]<(a:)]  =  ESiX[Xt]  =  $(t,s)x.  Write  Ff  for  the  right  continuous, 
complete  filtration  generated  by  <x{Xr  :  s  <r  <t},  and  Ft  =  P(°.  A  basic  result  (see  [8], 
[H])  is 


2 


,W.)  KW  *  (  'M  xl  ■’■rTfi,|i 1 


i 


LEMMA  2.1.  Mt  :=  Xt  —  Xq  —  /  ArXr-dr  is  an  {Ft}  martingale. 

Jo 

Proof.  Suppose  0  <  s  <  t.  Then 

E[Mt  -  Ms  \Fs}-E  [Xt  -Xs-  J  ArXr-dr  |  Fs 

=  E[xt-Xs-  J* ArXr-dr  \xi 
=  E[xt-Xs-  J*ArXrdr  |XS], 

because  Xr  —  Xr-  for  each  to,  except  for  countably  many  r, 

=  E.'XM  -X,-  J ArE,,x.{Xr)dr 

=  'i(t,s)X, -X,-  J  Ari(r,s)X,dr  =  0  by  (2.3). 


Therefore, 


Xt  =  X o+  f  ArXrdr +  Mt 
Jo 


=  Xo  -j*  f  ArXr-dr  +  Mt. 

Jo 

NOTATION  2.2.  If  x  —  (x0,  Xl5 . . .  ,xn)*  €  RN+l  then  diag  X  is  the  matrix 

/xo  0  ^ 


*1 


V 


0 


xN 


J 


We  now  give  a  martingale  representation  result. 
Lemma  2.3. 


X,  =  «((,0)(x0  +  f‘t(rAi)-'dMry 


(2.5) 


(2.6) 


Proof.  The  proof  follows  from  (2.5)  by  variation  of  constants.  Alternatively,  differ¬ 
entiate  (2.6). 


3.  Some  Basic  Martingales. 

If  x,  y  are  (column)  vectors  in  RN+1  we  shall  write  x  •  y  =  x*y  for  their  scalar  (inner) 
product. 

Consider  0  <  i,j  <  N  with  i  ^  j.  Then 
{Xs-  •  ei)e)dXa  =  ( Xs _  •  a)e) XX3 


=  (Xa-  •  ei)e*(X3  -  X3 _)  =  I(X3 _  =  ef,  =  e;). 


Define  the  martingale 


M”  ~  [\x3--ei)e*jc 

Jo 


(Note  the  integrand  is  predictable.)  Then 


M'tj  =  /*(*,_  •  -  f(X3-  ■  ei)e*AsXs-ds 

Jo  Jo 

and,  writing  Nt(i,j )  for  the  number  of  jumps  of  the  process  X  from  e,-  to  ej  up  to  time  t, 


this  is 


=  -  f‘l(X,.  =  e;)aji(s)ds 

Jo 

—  Nt(i,j)  f  I(X3  =  e,')aj;(s)c?s, 

Jo 


because  X3  =  X3-  for  each  oj,  except  for  countably  many  s.  That  is,  for  i  ^  j, 


)  =  fl{Xs  =  ei)aji(s)ds  +  M'tj. 
Jo 


For  a  fixed  j,  0  <  j  <  N,  write  Nt(j)  for  the  number  of  jumps  into  state  ej  up  to 


time  t.  Then 


N,(j)  =  Tt Mi,j)  =  E  /  I(X,  =  <iM») *  +  Mi 

i=i  t.iJ° 


N 

where  M/  is  the  martingale  )T) Mxt3 .  Finally,  write  Nt  for  the  total  number  of  jumps  (of 

i=i 

any  kind)  of  the  process  X  up  to  time  t.  Then 


N  N 


Nt  = 


=  EE  /  I(X,  =  efciMds  +  Q, 

i= 1 


N 

where  Qt  is  the  martingale  M{.  However,  from  (2.2) 

j=i 

N 

a»(s) =~Y1 

i=  i 

so 

=  I(XS  =  e.-K^Ks  +  Q«. 


(3.1) 


4.  Filtering. 

A  natural  problem  is  the  recursive  estimation  of  the  state  Xt  of  the  Markov  chain, 
given  the  number  of  jumps  which  have  occurred  to  time  t.  (Formulae  for  other  counting 
processes  such  as  Nt(j )  can  be  given.)  That  is,  we  have  on  the  probability  space  (ft,  F,  P) 
a 

SIGNAL,  given  by  (2.5), 

Xt  =  X o  +  /  AaXs-ds  +  Mt  (4.1) 

Jo 

and  an 

OBSERVATION  PROCESS,  given  by  the  counting  process 

Nt=  f* h(s,X3)ds  Qt.  (4.2) 

Jo 

Here 

N 

M«,x»)  =  -£jpr,  =  e,>«(4 
»=1 

Notation  4.1.  Write  a(s)  for  the  vector  (— aoo(-s),  —  an(s),...,  — aiv/v(s))*.  Then 
h(s,Xs)  =  a(s)  •  Xs.  We  shall  further  abbreviate  h(s,Xa)  as  h($). 

Recall  Xt  and  Nt  are  both  adapted  to  the  filtration  {i7)}.  Write  {Yt}  for  the  ri}  -it 
continuous,  complete  filtration  generated  by  A7,  so  Yt  C  Ft  for  all  t. 

NOTATION  4.2.  If  {^<},  t  >  0,  is  any  process  write  4>  for  the  F-optional  projection 
of  <f>.  Then,  (see  [7],  p.  60),  for  all  t  >  0 

}t  =  E[4t\Yt]  a.s.  (4.3) 
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Similarly,  write  4>  for  the  ^-predictable  projection  of  (f>.  Then,  for  allt  >  0 


h  =  E[<f)t  |  Yt-\  a.s. 


(4.4) 


Remarks  4.3.  From  Theorem  6.48  of  [7],  for  almost  all  w 

(f>t  =  <f>t 


except  for  at  most  countably  many  values  of  t.  Also,  as  noted  previously,  Xt  =  Xt~  except 
for  countably  many  values  of  t.  Therefore, 


Notation  4.4.  Write 

Pt  =  Xt  =  E[Xt  |  Ytl 

and  note  po  =  E[Xq)  =  po,  say.  Also,  as  h(r )  =  a(r )  •  Xr , 

h(r)  -  a(r )  •  Pr- 


We  shall  also  introduce  the  vector 


h{r)Xr  =  diag  a(r)  •  Xr, 
so  h(r)Xr  =  diag  a(r)  •  pr. 

Definition  4.5.  The  INNOVATION  PROCESS  is 

Qt  :=  Nt  —  f  h(r)dr 
Jo 

—  Nt—  I  h(r)dr  =  Nt—  I  h(r—)dr , 
Jo  Jo 


by  Remarks  4.3. 
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It  is  easily  checked  that  Q  is  a  Y -martingale.  Therefore,  we  can  write 


=  f  h(r—)d 

Jo 


r  +  Qt- 


Calculations  using  Fubini’s  theorem  show  that  the  process 


:=  pt-Po- 

Jo 


Aaps-ds 


is  a  square  integrable  martingale  on  the  Y -filtration.  Therefore,  M  can  be  represented  as 
a  stochastic  integral  with  respect  to  Q , 


=/ 


Mt=  7  rdQr 


where  7  is  a  T-predictable  RN+1  valued  process  such  that 


E[J~M*h(r)dr 


<  OO. 


Therefore,  we  can  write 


E[Xt  |  Yt]  =  pt  =  po  +  /  Arpr-dr  -f  /  7 rdQr. 

Jo  Jo 


It  is  known,  see  [4]  or  [5],  that 


7r  =  I(P(r- )  •  o(r)  ^  0)(p(r— )  •  a(r))  1  {diag  a(r)  -p(r-)  -  (p(r-)  ■  a(r))p(r-)  +  Arp(r-)j. 


Therefore  pt  —  E[Xt  |  Yt\  is  given  by  the  equation 

Pt  =  po  +  /  Arpr-dr  +  /  7r(dA^r  -  a(r)  •  pr_dr)  (4.8) 

Jo  Jo 

where  7 r  is  given  by  (4.7). 

REMARKS  4.6.  The  disadvantage  of  (4.8)  is  that  it  has  the  inverse  factor  (a(r)-pr-)_1  • 
This  problem  is  avoided  by  considering  the  related  Zakai  equation  for  the  unnormalized 
distribution. 
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Suppose  there  is  a  k  >  0  such  that  —aa(r)  >  k  for  ail  i  and  r  >  0.  Then  h(r)  1  = 
(a(r)  •  <  k~l  for  all  r  >  0.  Introduce  a  new  measure  P\  on  (£l,F)  such  that 


'[§'*]- 


where  A  is  the  martingale 


A<  =  1  +  f  Ar-(h(r-)  1  -  l)dQr. 
Jo 


(4.10) 


It  is  known,  (see  [4],  [5])  that  under  P\  the  process  Nt  is  a  standard  Poisson  process. 
Consider  the  (Pi, F)  martingale 


At  =  1  +  /  Ar_(/i(r— )  -  1  )dQ, 
Jo 


(4.11) 


Then  it  is  easily  checked  that  AtAt  =  1. 

To  obtain  the  Zakai  equation  we  take  Pi  as  the  reference  probability  and  compute 
expectations  under  Pi .  However,  it  is  under  measure  P  that 

Nt  =  f  h(r-)du  +  Qt. 

Jo 

Write  II(Af)  for  the  F-optional  projection  of  A  under  Pj.  Then  for  each  t  >  0 

n(At)  =  Pi  [At  |  Ft]  a.s. 

Quoting  again  from  [4]  or  [5]  we  know  that 

n(At)  =  1  +  f  A rdQr  where  Ar  =  H(A r-)(A(r-)  -  1).  (4.12) 

Jo 

For  any  integrable,  Pf-measurable,  random  variable  <f>  write 


Then 


Also, 


=  £.(A<#  |  Y,]. 


/  V  \  77*  fT  I  **  r  1 

cr(-rLtJ  =  -^U nt-o-t  j  it\  —  qt ,  say. 


r(l)  =  H(A,)  =  1  +  /  n(Ar-)(A(r-)  -  1  )dQ, 

J  0 


(4.13) 
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Now  qt  is  an  unnormalized  conditional  distribution  of  Xt  given  Yt,  because 


Pt  =  E[Xt  |  Yt]  =  a(Xt)/a(l)  =  qt/ H(At).  (4.14) 

Note  that  po  =  po  =  qo.  The  Zakai  equation  for  the  unnormalized  distribution  q  is, 
therefore,  obtained  by  calculating  the  product 

qt  —  Pt  ‘  n(A t)  (4.15) 

using  (4.8)  and  (4.13)  to  obtain:  * 

qt  =  $o  +  /  Arqr-dr  +  [  (diag  a(r)  -  I  +  Ar)qr-dQr.  (4.16) 

Jo  Jo 


5.  Optimal  Control. 

Consider  a  Markov  process  X ,  as  defined  in  Section  2,  whose  state  space  is  the  set  S 
of  unit  vectors  {e0,  ei, . . . ,  e#}  of  RN+1 .  However,  we  now  suppose  that  the  family  of  Q- 
matrix  generators  At(u)  depend  on  a  control  parameter  u  EU.  Here  U,  the  set  of  control 
values,  is  a  compact,  convex  subset  of  some  Euclidean  space  Rk.  We  take  0  <  t  <  T  and 
suppose  At(u )  is  measurable  on  [0,T]  x  U  and  A<(-)  is  continuously  differentiable  on  U. 
Further,  we  assume  |a,y(f,  it)|  <  B  for  (t,  u)  G  [0,T]  x  U. 

We  suppose  that  only  the  total  number  of  jumps  to  time  t,  Nt ,  is  observed.  (The 
techniques  below  work  for  other  kinds  of  observation  processes  such  as  Nt(j ,  k),  Nt(j )  etc.) 
{Ft},  (resp.  {Ft}),  is  the  right  continuous,  complete  filtration  generated  by  X  (resp.  N). 

CONTROLS  5.1.  The  set  j7  of  admissible  controls  is  the  set  of  {yj-predictable 
processes  with  values  in  U.  This  means  that,  if  Ti,T2,  . . .  are  the  jump  times  of  N,  then 
for  T„  <t  <  Tn+ 1 ,  u  €  U_  is  a  function  only  of  T\ ,  T2 , . . . ,  Tn  and  t.  For  each  u  G  U,  as  in 
Lemma  2.1,  Mu  is  a  (P,  {Pt})  martingale  where 


Mu 


—  vu 

.—  -- j  - -u 


_  r 

Jo  ‘ 


(5 


1  \ 


For  u  G  U_,  write 


a(s,u)  =  (-a0o(s,u),  -an(s,u),...,  -aNN(s, u))* 
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and 

h(s,u)  =  h(s,X!)  =  a(s,u)-X;.  (5.2) 

Then  with  ~  again  denoting  the  F-optional  projection  under  P 

?.(«)  =  x;  =  eix?  |  y.) 

and  h(s,u)  =  a(s,u )  •  ps(u).  Also, 

h(s,u)-X?  =  diag  a(s,u)-X“ 


Nt  can,  therefore,  be  written 

Nt=  [  h(s,u)ds  +  Q “  (5.3) 

Jo 

h(s—,  u)ds  +  Q“.  (5.4) 

Unlike  the  situation  considered  in  [9],  the  noises  in  the  signal  and  observation  processes 
are  correlated,  [-AT u ,  JV] <  ^  0. 

COST  5.2.  A  real  function  on  the  state  space  S  =  {eo,  e\, . . . ,  e,v}  is  represented  by 
a  vector  £  =  (£0,.. .  ,£n)*  €  RN+1.  Write  (£,x)  =  £*  •  x  for  the  inner  product  on  RN+1. 
The  control  problem  we  wish  to  consider  is  that  of  choosing  u  €  H  so  that  the  expected 
cost 

J{u)  =  E[(e,  Jff)] 

is  minimized. 

Notation  5.3.  Write  Pi,  as  in  Section  4,  for  the  probability  measure  under  which 
N  is  a  standard  Poisson  process. 

For  each  u  €  IL  introduce  the  (Pi,P)  martingale 

A?  =  l+  fVr_(h(r-,u)-l)iQr 
Jo 
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and  write  II(A^)  for  the  ^-optional  projection  of  A  under  Px.  Then,  E\  [jjjr  i  -pi]  = 
and  with 

»W)  =  •EiE‘X,“  |  Yt)  =  s,(«i) 

we  have,  as  in  (4.15), 

qt{u)  =  U(K^)pt(u). 

Furthermore,  with  At(u)  =  (a,-;-(t,u))  write 

Bt(u)  =  (diag  ( a(t,u ))  -  /  +  At(u)) 

/  -1  a0i(t,u)  ...  a0N(t,u)\ 

a10(t,u)  -1  ...  alN(t,u) 


\aNo(t,u)  .  -1  / 

Then,  for  u  GU,  the  unnormalized  distribution  qt(u)  is  given  by  the  Zakai  equation 

qt(u)=p0+  f  Ar(u)qr..(u)du  +  [  Br(u)qr_(u)dQT. 

Jo  Jo 

Here  Q  is  a  standard  Poisson  process  under  P\ . 

The  expected  cost  if  u  6  ££  is  used  is 

J(u)  =  E[{ltXfi)  =  E^Vt^X})]  =  E1[{£,  VTXfi) 


=  PX[</,PX[ VtX$  I  Yt)))  =  &[&&)). 

The  control  problem  has,  therefore,  been  formulated  in  separated  form:  find  u  €  U_  which 
minimizes 

J(u)  =  E1[(£,q^)) 

where  for  0  <  t  <  T,  qt(u )  satisfies  the  dynamics 

2t(w)=Po+  f  Ar(u)qr-(u)du  +  f  Br{u)qr-(u)dQr.  (5.5) 

Jo  Jo 

Under  the  measure  Px,  N  is  a  standard  Poisson  process,  so  Qt  =  Nt  —  t  is  a  (Px,y) 
martingale.  Furthermore,  the  controls  u  6  U  are  in  the  ‘stochastic  open  loop’  form  dis¬ 
cussed  by  Bismut  [2]  and  Kushner  [10].  That  is,  the  controls  are  adapted  to  the  filtration, 
as  described  above,  and  are  not  explicitly  functions  of  the  state  q. 


6.  Differentiation. 


NOTATION  6.1.  For  u  €U  write  $u(t,s)  for  the  fundamental  matrix  solution  of 

d$u(t,.s)  =  At(u)^'t(t-,s)dt  +  Bt(u)$v(t-,s)(dNt  -  dt)  (6.1) 

with  initial  condition  $u(s,  s)  =  I,  the  (N  +  1)  x  (N  +  1)  identity  matrix. 

Note  that  At(xi)  -  Bt(u )  =  diag  (1  +  aa(t,u))  and  write 

Du(s,t)  =  diag  (exp  J*  (1  +  a,-,-(r,u))dr). 

Then  if  Tn  <  t  <  Tn+1, 

$“(t,0}  =  Du(t,Tn)(/+Srn(urJ)D«(Tn,Tn_1)(/  +  BTo.l(uTn.J) 

...(I  +  BTl(uTl))D'{Tu  0).  (6.2) 


The  matrices  Du(s,t )  have  inverses 

diag  (  exp  -J  (1  +  ai{(r,  u))drj ; 
we  make  the  following  assumption: 

Assumption  6.2.  For  u  e  V.  and  <  G  [0,T]  the  matrix  (I  +  Bt(ut ))  is  nonsingular. 
The  matrix  $  is  the  analog  of  the  Jacobian  in  the  continuous  case.  We  now  derive 
the  equation  satisfied  by  the  inverse  $  of  $. 


LEMMA  6.3.  For  u  €  V_  consider  the  matrix  defined  by  the  equation 

Vu(t,s)  =  I-  j:  \Pu(r-,  s)Ar(u)dr  -  J  ^“(r— ,  s)Br(u)dQr 
+  jf  V(r— ,  s)B2r{u)(I  +  BrW'dNr. 

Then  \i/u(t,s)$u(i,s)  =  I  for  t  >  s. 


(6.3) 


Proof.  Recall 


#u(i,s)  =  1+  J  Ar(u)$“(r-,s)dr  +  j*  Br(u)^xl(r-,s)dQr.  (6.4) 
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Then  by  the  product  rule 


=  1+  ^  $A$dr  +  J  ^B^dQr 

-  f  <$A$dr-  f  VB$dQr+  /  ^B2(I  +  B)~l$dNr 

Js  Ja 

ft  ft 

-  j  %B2$dNr  +  J  VB2(I  +  B)-1B$dNr 


as  the  integral  terms  cancel.  □ 

We  shall  suppose  there  is  an  optimal  control  u*  £  U.  Write  q*  for  qu  ,  $*  for  4>u 
etc.  Consider  any  other  control  v  £  U_ .  Then  for  8  £  [0, 1], 

ug(t)  =  u*{t)  +  8(v(t)  —  u(t ))  £  U_. 

Because  U  C  Rk  is  compact,  the  set  U  of  admissible  controls  can  be  considered  as  a 
subset  of  the  Hilbert  space  H  =  L2[fl  x  [0,T] :  Rk\.  Now 

J(u  >  J(u*).  (6.5) 

Therefore,  if  the  Gateaux  derivative  of  J,  as  a  functional  on  the  Hilbert  space  H, 

is  well  defined,  differentiating  (6.5)  in  9 ,  and  evaluating  a.t  9  =  0,  implies 


{J'(u*),v(t)-u*(t))  >0 


for  all  v  €  IL- 


LEMMA  6.4.  Suppose  v  £  U_  is  such  that  u*6  =  u*  +  9v  £  JJ_  for  9  £  [0,  a].  Write  qt(8 )  for 

iL.  _ _  .r  /er  ml _ _  dqt(9')\  _  _ i  •  _ : _ ./jL. 


the  solution  qt(u*g )  of  (5.5).  Then  zt  = 
equation 


exists  and  is  the  unique  solution  of  the 


zt  =  j  (^“(r)u*))urg*_dr  +  J  Ar(u*)zr-dr 

+  J  (^~(r>  u*i)vrqi-dQr  +  J  Br(u*)zr-dQr.  (6.6) 

rl  .  rl  - 

Proof.  qt(8)=po+  I  Ar(u' +9v)qr-(8)dr+  I  Br(u* +  9v)qr-(8)dQr.  The stochas- 

J  o  Jo 

tic  integrals  are  defined  pathwise,  so  differentiating  under  the  integrals  gives  the  result. 
Comparing  (6.4)  and  (6.6)  we  have  the  following  result  by  variation  of  constants. 


Lemma  6.5.  Write 


m,,  =  jjV(r-,0)(|^(r,o)<V?;-<fr 

-  r«>(r-,0)(J  +  Br(U'))-1Br(»-)(|?(r,tt*))1;rg;_</JVr.  (6.7) 

Then  zt  =  $*(£,0)770,,. 

Proof.  Using  the  differentiation  rule 

$*(*>0)770,,=  /V_-d7?+ 


Because  =  I,  therefore 


$*(*>0)t7o,,  =  J  (|^(r,u*)jurg;_dr 

-  jf  (7  +  Br{u*))-lBr{u*)  (|^(r,  «•)) vrq*_dNr 

+  f  J4r(n)$*(r—,  0)770, r-dr  +  f  Br(u)$*(r~,  0)770, r_dQr 
do  do 

+  J  Br(u)(^(r,u'fjvrq'_dNr 

-  J  Br(u){I  +  B.OO)’1  Br(«*) (^(r,  u‘))vrq‘_dNr. 

Now  the  diV  integrals  sum  to  0,  showing  that  $*77  satisfies  the  same  equation  (6.7)  as  z. 
Consequently,  by  uniqueness,  the  result  follows. 


flAnAt  *  *  »>*» 


/>  /? 
o.u. 


dJ, 


Wu»>\e=l 


n  r//i 

=  ^,uj77o,r;j- 


Proof.  J(ug )  =  Tu  [{^,  $r(0)  >].  The  result  follows  from  Lemmas  6.4  and  6.5. 
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NOTATION  6.7.  Write  $*(T,  0)'  for  the  transpose  of  3>*(T,  0)  and  consider  the  square 
integrable,  vector  martingale 

Mt  :=£1[<r(T,0)'£|  Y,]. 

Then  Mt  has  a  representation  as  a  stochastic  integral 

Mt  =  E1[$*(T,  0)T]+  f\rdQr 

Jo 

where  7  is  a  predictable  RN+1  valued  process  such  that 

Ei[y^\dr  <  00. 

Under  a  Markov  hypothesis  7  will  be  explicitly  determined  below, 

DEFINITION  6.8.  The  adjoint  process  is 

pt  := 


Theorem  6.9. 

rT 


d8 


=  l  El  [(Pr-,  {  (^(r,«*)  -  (/+  Br(u-)r'Br(u'){^(ryj) 

+  (Tr,  **(>•-, 0)(J  +  Br(u-))-'  (|?(r, U*))»r?;.)]  dr. 


(6.8) 


Proof.  First  note  that 

{ MT,m,T )  =  J  ( Afr_,  ^(r-,  0) (y~(r> u'))vr'i-jdr 

+  j  {Mr.y(r-,f))(~(ry))vrq-.)iQr 

/*T  .  »T  **  -% 

+  J  (lnVo,r-)dQr  +  ^  (7r#*(r~ 0) (~-(r, «*)) vrq*rJ}dNr 
-  £(yr,  'r(r-  0 )(/  +  Sr(M*))- xBr(tO  (~(r,  u*f)vrq*r_)dNr. 
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Taking  expectations  under  P,  we  have 

d9  6=o 


=  E1[(£,$'(T,  0)rjQiT)} 

=  Ei[(MT,r]oyT)]- 


Combining  the  last  two  terms  in  (6.9)  and  using  the  fact  that  Nt  —  t  is  a  Pi  martingale, 


this  is 


-(pr.,(J  +  Br(U*))-‘Br(u*)(|?( 

+  (7r,  f*(r-,  0 )(/  +  B,(n-)YX  (^(r,  u*))<vs;_)]<fr. 


Now  consider  perturbations  of  u*  of  the  form 


ug(t)  =  u*(t)  +  9(v(t)  -  u(t )) 
for  9  £  [0, 1]  and  any  v  £  U,  Then  as  noted  above 


dj(ug ) 
d9  e=o 


=  <jV)M0 -«*(*))  >0. 


Expression  (6.8)  is,  therefore,  true  when  v  is  replaced  by  v  —  u*  for  any  v  £  U,  and  we  can 
deduce  the  following  minimum  principle. 

THEOREM  6.10.  Suppose  u*  £  TJ_  is  an  optimal  control.  Then  a.s.  in  w  and  a.e.  in  t 

(?> — .  {  (f)(r.O  -  (I  +  Br(«*))-‘Br(U-)(|^(r,  «*))  }(»,  -  <)«;.} 

+  /7„  .p*(r-,0)(J+  B^u'))-1  (|^(n«‘))(»r  -  «»?;-)  >  0.  (6.10) 
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7.  The  Equation  for  the  Adjoint  Process. 

The  process  p  is  the  adjoint  process.  However,  (6.10)  also  contains  the  integrand  7.  In 
this  section  we  shall  obtain  a  more  explicit  expression  for  7  in  the  case  when  u*  is  Markov, 
and  also  derive  forward  and  backward  equations  satisfied  by  p. 

ASSUMPTION  7.1  The  optimal  control  u*  is  a  Markov,  feedback  control.  That  is, 
u*  :  [0,T]  x  RN+1  — »  U  so  that  u*(s,q*_ )  £  U. 

Note  that  if  um  is  a  Markov  control,  with  a  corresponding  solution  qt(um )  of  (5.5), 
then  um  can  be  considered  as  a  stochastic  open  loop  control  um(u>)  by  setting 

um(uj)  =  um(s,q*_(um)(u)). 

This  means  the  control  um  “follows”  the  ‘left  limit’  of  its  original  trajectory  q3(um )  rather 
than  any  new  trajectory. 

LEMMA  7.2.  Write  6  for  the  predictable  “integrand”  such  that 


Apt  =pt-  Pt-  =  A Nt, 


i.e.,  pt  =  pt _  +  StANt. 


Furthermore,  write 


qt~  =  q, 

Bt-(u*(t-,q))  =  B*{qt-)  =  B*(q] ), 


and 


Then 


St(q)  =  (7  +  B*((/+B*(f))s))-V<-(U  +  ir(5))«) -?,-(«). 


(7.1) 


Proof.  Let  us  examine  what  happens  if  there  is  a  jump  at  time  f;  that  is,  suppose 
ANt  =  1.  Then  from  (5.5) 

q,=(I  +  B-(q))q. 


sasfcffi 
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By  the  Markov  property  and  from  (6.2)  and  Definition  6.8, 


P,  =  E[D'(t,Tk)(I  +  B'n(u')).. . D-(T,TN)l  |  7,] 

=  P,((I  +  B\q))q) 

=  (J  +  J‘WrV((f  +  «'(lM 

and  the  result  follows.  Heuristically,  the  integrand  S  assumes  there  is  a  jump  at  i;  the 
question  of  whether  there  is  a  jump  is  determined  by  the  factor  A iV*. 

THEOREM  7.3.  Under  Assumption  7.1  and  with  8t  given  by  (7.1) 

1r  =  $*(r--,0 )'((/  +  B'r(u*))6r  +  B'r(u*)Pr-).  (7.2) 

Proof.  $*(t,0)'pt  =  Mt  =  E![$*(T,0)^  |  Yt]  =  ^‘(T.O)'*]  +  f  \rdQr.  However, 

Jo 

if  u*  is  Markov  the  process  q*  is  Markov,  and,  writing  q  —  q*,  $  =  <&*(<,  0), 

^[$*(T,0)'£  |  Yt)  =  |  q,$) 


=  $'E1[$*(T,t)'e\q). 

Consequently,  pt  =  Ei[$*(T,  t)'£  |  q]  is  a  function  only  of  q ,  so  by  the  differentiation  rule: 
_ ,  ttdPr-,  .  ,  n.  jTSn  ,  ftQPr-  , 


Pt  =  Po  + 


fldpr- 

Jo  dq  {Aqr- 


dr  +  Bqr-dQ. 


£( 

0<r<t 


Pr  -  Pr-  ~ 


-dQr)  + 1  a 

■B?r_AiVr) 


=  Po  +  J  -~(Aqr-  -  Bqr~)  +  <5r]  dr  +  J  8rdQr. 


Evaluating  the  product: 


=  $*(t,  0)'pt  =  po  +  J  (r-,  0)'  [^(Agr_  -  B?r_)  +  5r]  dr 

ft  _  /*< 

+  J  <F(r-,0)/-~—  dr  +  J  $~(r-,Q)!8rdQr  +  j  #*(r-,G)'A'pr-aV 

+  /V(r-,0)'.B'pr-dQr  +  f$*(r-,0yB'8rdQr+  f  $*(r-,0)'  B'8rdr. 
Jo  Jo  Jo  I 
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However,  Mt  is  a  martingale,  so  the  sum  of  the  dr  integrals  in  (7.3)  must  be  0,  and 

7r  =  $*(r-,0)'(*r  +  B'r(u*r)6r  +  B'r(u*r)pr-). 


THEOREM  7.4.  Suppose  the  optimal  control  u*  is  Markov.  Then  a.s.  in  <x>  and  a.e.  in  t, 
u*  satisfies  the  minimum  principle 

(pr-,  -  u‘)qU)  +  {«,,  ^(r,u*)(»r  -  <)s;_)  >  0.  (7.4) 

Proof.  Substituting  7  from  (7.2)  into  (6.10),  and  noting  B(I+B)~l  —  (I+B)~1B  =  0, 
the  result  follows.  (Substituting  for  B  and  8  gives  an  alternative  form.) 

We  now  derive  a  forward  equation  satisfied  by  the  adjoint  process  p : 

Theorem  7.5.  With  8  given  by  (7.1) 


Pt  =  ExmT,0ye}-  f  A'r(u*r)pr-dr 

Jo 

-  f\l  +  B'r(u*r))8rdr  +  fsrdNr. 
Jo  Jo 


Proof,  pi  =  tf*(t,0)'Mt  and  from  (6.3)  this  is 

=  Ei[$*(T,0)'2]  -  f  A'WMdr-  f* B'V*,MdQr+  f  (I  +  B'Y'B'2®*' MdNr 

Jo  Jo  Jo 

+  ft'b*'A(rdQr  —  f  B'VirdNr  +  f(I  +  B,)-lBn%*,~trdNr 

Jo  Jo  Jo 

=■-^[$*(7,0)^-  f* A1  pr-dr  -  t B'pr-dQr  +  f\l  +  B')~l  B'2pr-dNr 
Jo  Jo  Jo 

+  [ \(I  +  B')8r  +  B'pr-)dQr  -  f  (I  +  B')~lB'((1 4-  B')8r  +  B'pr-)dNr 
Jo  Jo 

=  jEJi($*(T,0)'£j-  f  A'pr-dr  +  f  ( I  +  B')6rdQr  -  f  B'8rdNr 

Jo  Jo  Jo 

and  the  result  follows.  D 

However,  an  alternative  backward  equation  for  the  adjoint  process  p  is  obtained  from 
the  observation  that  the  sum  of  the  bounded  variation  terms  in  (7.3)  must  be  identically 
zero.  Therefore,  we  have  the  following  result  which  appears  to  be  new: 
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THEOREM  7.6.  With  6  given  by  (7.1)  the  Markov  adjoint  process  pt(q)  is  given  by  the 
backward  equation 

^  •  (A‘(iY  -  B‘{qy)q  +  A‘(q)'p,  +  (I  +  B*(q)')St  =  0 

with  the  terminal  condition 

PT  =  £ 


8.  Conclusion. 

A  finite  state  space  Markov  chain  was  considered.  Without  loss  of  generality  its 
state  space  was  taken  to  be  the  set  of  unit  basis  vectors  of  RN+1.  Basic  martingales 
associated  with  the  Markov  chain  were  identified  and  the  solution  to  filtering  problems 
given  when  only  the  total  number  of  jumps  are  observed.  On  the  basis  of  knowing  only  the 
total  number  of  jumps  a  control  problem  associated  with  the  Markov  chain  is  discussed 
in  ‘separated’  form.  That  is  the  Zakai  equation  for  the  unnormalized  distributions  is 
obtained.  This  is  a  linear,  vector  equation  driven  by  a  standard  Poisson  process  in  which 
(unlike  earlier  work  on  controlled  Markov  chains)  the  control  variable  also  appears  in  the 
‘diffusion’  coefficient  multiplying  the  noise  term.  By  adapting  techniques  of  Bensoussan 
and  calculating  a  Gateaux  derivative  the  minimum  principle  satisfied  by  an  optimal  control 
is  obtained.  Finally,  in  the  case  when  the  optimal  control  is  Markov,  the  integrand  in  a 
martingale  representation  can  be  obtained  explicitly,  and  forward  and  backward  equations 
satisfied  by  the  adjoint  process  derived. 
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ABSTRACT:  By  considering  small  perturbations  in  time,  which  are  then  com¬ 
pensated  by  changing  the  measure,  a  new  integration- by-parts  formula  is  obtained  for 
functionals  of  a  single  jump  process.  For  martingales  associated  with  observing  the  time 
of  the  jump  a  new  expression  is  derived  for  the  integrand  when  they  are  represented  as 
stochastic  integrals. 


W. .1-  if  "W  h 


Integration  by  Parts  for  the  Single  Jump  Process 
Robert  J.  Elliott  and  Allanus  H.  Tsoi 

1.  Introduction. 

Integration  by  parts  has  played  a  basic  role  in  the  Malliavin  calculus  and  its  applica¬ 
tions,  particularly  in  the  work  of  Bismut  (1981).  In  this  paper  the  concept  is  investigated 
in  the  fundamental  situation  of  a  stochastic  process  with  a  single  random  jump.  When  the 
state  space  of  the  process  is  Euclidean  space  (or,  possibly,  an  open,  non-empty  subset  of 
a  Euclidean  space)  the  techniques  of  Norris  (1988)  can  be  specialized  to  the  single  jump 
situation.  This  method,  described  in  Section  2,  considers  a  small  e-perturbation  in  the 
state  space  of  the  process.  The  effect  of  the  perturbation  can  be  removed  by  a  Girsanov 
change  of  measure,  and  the  integration  by  parts  formula  is  obtained  by  differentiating  in  e. 

However,  for  a  process  whose  state  space  is  a  general  measure  space,  the  perturbation 
of  the  kind  considered  by  Norris  may  not  make  sense.  Such  processes  include  those  with 
discrete  state  spaces,  and,  in  particular,  the  process  which  observes  a  single  random  instant 
at  a  time  T.  In  the  latter  case  the  process  p<  =  It>r  takes  only  the  values  0  or  1. 

For  general  jump  processes,  therefore,  an  alternative  e-perturbation  in  the  time  direc¬ 
tion  is  introduced.  By  differentiating  a  new  integration  by  parts  formula,  which  involves 
a  time  derivative,  is  obtained.  In  the  case  of  the  fundamental  process  p<  an  alternative 
expression  for  the  integrand  in  a  martingale  representation  result  is  derived. 

2.  Integration  by  Farts  for  IR^- Valued  Single  Jump  Processes. 

Consider  a  single  jump  process  with  state  space  jtRd  for  some  d  >  1,  which  remains  at 
its  initial  position  until  a  random  time  T,  when  it  jumps  to  a  new  random  position  Z. 
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The  underlying  probability  space  is  taken  as  ([0,cc]  x  HI'*,  #((0,oo])  x  B(lRti),p).  For 
t  >  0,  let  be  the  completed  er-field  generated  by  the  process  up  to  time  t.  Suppose 
(A,  A)  is  the  Levy  system  for  the  process  (see  Elliott  (1982)).  For  A  6  ^(IR^),  let 


p{t,A)  =  It>rIzeA 

(2.1) 

p{t,A)  =  -  f  A(s,A)~ 

JjO.tAT]  *!>- 

(2.2) 

where  Ft  =  /j(]i,oo]  x  IR<().  Then  q(t,A )  =  p(t,A )  -/>(t,A)  is  an  ft -martingale. 

We  assume  that  Ft  and  A  are  absolutely  continuous,  so  that  there  exist  functions  f3 
and  g(y)  >  0  such  that 


dF,  —  f $ds 
Hs>dy)  =  g{y)dy. 


Consequently, 

[  ~9(v)  dyds  if s  <T 
P(ds,dy)  =  < 

l  0  if  $  >  T. 

Let  v(t,  y)  be  an  IR^-valued  function  which  satisfies: 

(i)  v(t,  •)  is  Cl  for  each  t  >  0;  v  and  v(t,y)  are  uniformly  bounded. 

(ii)  supp  u(-,  •)  C  (0, oo )  x  I(  for  some  compact  K  C  IR^. 

For  small  z  €  IR  and  any  (j>  6  L*(p),  define  p£  by: 

/  [  <f>(s,y)pe(ds,dxj)=  f  f  <f>(s,6c(s,y))p{ds,dy), 
Jo  Je  J»  Jg 

where 

9e(t,y)  =  y  +  zv(t,y). 


(2.3) 


(2.4) 
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Set 


A  c(t,y)  = 


90£(t,y)  gjF&y)) 
dy  <)(v) 


and 


Xt  =  f  [  (Xs{s,y)-l)q(ds,dy). 

Jo  Je 

Define  the  family  {Zf ,  t  >  0}  of  exponentials  by: 

Z;=acp(X,-i<XVY‘)l)  J]  (1  +  AA',)c-aa'- 

~  0<s<t 


Then  Zf  satisfies: 


=  exp  ( J  J^\ogXe(s,y)dp-  J  J^(Xc(s,y)  -  l)dp). 

Zf  =  f  f  Z‘3_(X‘(s,y)-l)q(ds,dy) 

Jo  Je 


and  {Zf,  t  >  0}  is  a  martingale  with  E[Zf]  =  1. 
Define  a  new  probability  measure  p‘  by: 


dp  rye  *t~ 

-7—  =  Z,  on  Tt. 
dp 


Lemma  2.1.  Under  p£,  pc  has  the  original  law  of  p. 


Proof.  It  suffices  to  check  for  test  functions  <£.(=  Ll(p)  and  for 


Vf  =  exp  |  J  J^(s,y)pe{ds,dy)}zf 
=  exp  {J  jj>{s,e€(s,y))p{ds,dy)}zf 


that  E[Uf)  does  not  depend  on  e.  Let 


V,  =  exp  {J  jji(.‘,,eU3,yj)p(ds,dy)j. 


(2.5) 

(2.6) 

(2.7) 

(2.8) 
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By  the  differentiation  rule, 


But 


Jo  Je2*-^*  Jo  ^,^-fexp(^(5>^(5.J/)))  ~  l]p(cfe,cfy) 

AK,  =  YT-{exp{ct>(T,8£(T,Z))}  -  ljjs=r 
AZt=Z^(T,Z)-l)I3=r. 


Hence, 


[Y,Zs)t  =  AKrAZf/,>r 


=  ^’f-(exp{^(T,  9e(T,  Z ))}  -  1][AC(T,  Z)  -  lj/t>r 
=  l  Jvi-lextMs, <>•(*, y))}  -  l](A«(s,j,)  _  1  }p(ds,dy). 


Hence, 


U;  =  1  +  MartinSal°  +  l  Jw-fr  PM*,  «•(», »))}  -  l]A£(s,  y)p(dS,  dy) 

=  1  +  Martingale  +  fJu‘,-bM^‘(s,y)))  -  l)\’{s,y)p(ds,iy) 

=  1  +  Martingale  -  J  j  u^oxp^eHs.y))}  -  1]A£(S,  y)y(y)  L  dyds. 

* 3 


Thus 


rl  r 


EWl)  -  1  -jo  JLEiU;)[exp{<}>(s,Oe(S,y))}  -  l]^(a,y))  L  dyds 

~  1  ~  JQ  JeEWs\[cx p{^(5)J/)}  -  1  \g(y)  —■  dyds 
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by  the  Jacobian  formula.  Thus  E[Uf]  is  independent  of  e. 


As  a  consequence  of  Lemma  2.1,  we  have 

E[Zct exp{<j>(T,  2  +  eV(T,  Z ))}]  =  £[exp{^(r,  2 )}]  (2.9) 

which  leads  us  to  the  following  theorem: 

THEOREM  2.2.  Suppose  G  :  (0,oo]  x  IRd  — *■  IR  is  positive,  bounded  and  that  its  partial 
derivative  — — -  exists  and  is  bounded.  Then 

Bl(f0  /E(|  +  !/))?(*, <w)g(t,z)]  =-E\^§~1  V(T,Z)  . 


Proof.  Differentiate  (2.9)  with  respect  to  e ,  then  set  e  =  0  to  obtain 


|„0^<WZ+eV(T,Z))}|J 

+eM  n  Tc  exp {<f>(T,Z  +  eV(T,Z))}  |  =  0. 

*•  l£=0  «£  £=oJ 


From  (2.S), 


(2.10) 


(2.11) 


*  Z‘T  ~  l  JE  ^r(A'(‘>!')  -  ^Wt.dy) 

f'l  r  ^ 

+  i)  //'-  de 


From  the  definition  of  \e(t,y), 


\cd 


and  from  (2.7), 


Also 


d\£(t,y) 


(l£ 


=  jr  v{t,y)  +  ^rr 
*=o  Oij  g[y) 


Hence 


±z* 

ds  Zt 


~o=l  JSijv(t'y)+a-Mv(t'y)^(dt'dy)- 


Thus  (2.11)  becomes 


E 


(j(  fjSy  +  i/j) •?(«*■  <>w)  e*pW(2\  Z)l] 

=,-B[exp{MT,Z))&«T,Z))viT,zj\.  (2.12) 


Let  <p(T,  Z)  =  log  G(T,Z).  Then  (2.12)  becomes  (2.10)  and  the  proof  is  complete.  □ 


3.  Integration  by  Parts  for  a  General  Jump  Process. 

Consider  a  single  jump  process  with  values  in  a  Lusin  space  (E,  £).  The  underlying 
probability  space  is  ([0,oo]  x  E ,  £5([0,  oo])  x  £,  fi).  In  this  section  we  suppose  that  for 
every  t  >  0,  Ft  >  0,  and  both  Ft  and  A*  axe  continuous  in  t.  Furthermore,  we  assume 
that  there  exists  a  function  a •($),  with  a(s)  >  0  for  all  s  >  0,  such  that 

At  =  /  a($)ds. 

Jo 

Let  u  :  [0,  oo]  x  E  — *  HI  be  a  bounded,  positive,  deterministic  function  such  that 

u,(y)  =  0  if  s£[ 0, 6] 

for  some  fixed  b  €  IR.  For  e  >  0,  define 

A ?  =  /  /(I  +eu3(y))\{s,dy)dAa.  (3.1) 

Jo  Jb 

G 


Consider  the  new  measure  /xe  which  has  a  Levy  system  (A,A£).  Then  (see  Elliott  (1982)) 


/C  <  /i,  and  if 


■«  d^c 


we  have 


LCW  ~  f£(l  +eut{y))Kt,dy)exp{  ~  J  Jeu,(y))\{s,dy)dAsy 


Furthermore,  if  L\  =  E[L'{t)  |  Ft)  then  {!',  t  >  0}  satisfies 


Lct=  1  +  flca-dMs 

Jo 

=  1  +  /  [  £Us(y)Ms,dy)q(ds,E), 

Jo  Je 


where 


Mt  -  I  f  £uo(y)Ks^yk(ds,E). 

Jo  Je 


If  Ff  =  /i£(]t,oo]  x  E),  then 


Define 


Ff  =  Ftexp{-J  J  £us(y))X(s,dy)dA3y 


&(0  =  sup{s  :  F‘  >  Ft}. 


Then  4>e(t)  is  an  increasing  function  of  t,  and  F £  ^  -  Ft,  i.e., 

A**(hk(t)>  o°]  x  E)  =  n{)t,oo)  x  E). 

Hence  if  we  let  <f>e(t)  =  ^  J(t),  then  under  /H,  0e(T)  has  the  same  distribution  as  T 
under  ft.  This  observation  leads  us  to  the  following  theorem: 


THEOREM  3.1.  Let  G(t,z)  be  a  real-valued  function  defined  on  [0,oo]  x  E,  which  is 

r\ 

bounded  and  has  bounded  partial  derivative  j^G{t,z).  Then 


E 


-E 


p 

Oo  j<y)x^dy)^E^T^\ 

J  J  ut(y)\(t,dy)atdt\. 


dG(T,Z)  1 


dt  a(T) 


(3.5) 


Proof.  Prom  the  above  discussion  we  have 


e{G(t,z)]  =  e‘IG(mt),z)} 


=  E\L‘tG(^(T),Z)) 


(3.0) 


where  Ec  denotes  that  expectation  is  taken  with  respect  to  pe.  Differentiate  (3.6)  with 
respect  to  e,  then  set  e  =  0  to  obtain. 


E 


dLfr 


l  de 


«=o 


G(MT),Z) 


c=0J 


+  E 


Lt 


-rG(Mnz) 

c=o  ds 


€=0 


=  0.  (3.7) 


From  (3.2)  and  (3.3), 


dLe, 


de 


p 

=  [  [  «t(y)A(f,dy)g(di,£). 

«=°  Jo  Je 


Also, 


±G(UnZ)\^LG(T<Z)lMT) 


£  =  0 


To  evaluate  -^z  4>t{T)  .,  note  that  F £i(()  =  Ft.  Hence 

rt  r 

Ft  =  F^i)  ~  Ft  exp  j  -  /  /  £uJ(y))A(s,dy)dA,J 

J  0  J  Ly 

=  Ft(~  ft  u3(y)\(s,  dy)dA,) . 
de  £=o  '  Jo  Je  ' 


(3.8) 


(3.9) 
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where  u  is  just  a  function  of  the  time,  which  is  positive,  bounded  mid  vanishes  outside 
a  bounded  interval,  then  the  method  described  in  Section  3  would  give  us  the  simpler 
integration  by  parts  formula: 


rp 

e[(J  nA)G(T)]  =-e[ 


dG(T)  1  fT 

dt  a(T)  J0 


usa3ds 


where  G  is  a  bounded  function  defined  on  (0,  oo]  with  bounded  derivative.  On  the  other 
hand,  if  we  assume  i?[<j(T)]  =  0,  then  G(T)  has  the  martingale  representation  (see  Elliott 
(1982)): 

G(T)=  [r^dq3  (4.2) 

Jo 

where 

7,  =  G(s)  -  F~l  f* G(v)dFv. 

Jo 

If  we  substitute  (4.2)  into  the  left  side  of  (4.1),  we  have 


u3dq3 


f  Isdqjj  =  E  J  u3'y3d(q,q)9\ 

=  E^  Us73a3ds 

vO 


r  CO 

=  E  I3<t  u373a3ds  . 


Now,  if  we  define  the  measure  tt  by: 


*(dt)  =  SjTj  5r{it)' 
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then  the  right  side  of  (4.1)  is 


11 
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Abstract.  The  optimal  control  of  a  partially  observed  diffusion  is  discussed  when  the 
control  parameter  is  present  in  both  the  drift  and  diffusion  coefficients.  Using  a  differen¬ 
tiation  result  of  Blagovescenskii  and  Freidlin,  and  adapting  techniques  of  Bensoussan,  a 
stochastic  minimum  principle  is  obtained. 
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1.  Introduction 


The  adjoint  process,  and  related  minimum  principles,  for  partially  observed  stochastic 
control  problems  have  been  investigated  in  several  recent  papers.  See,  for  example,  the 
works  of  Bensoussan  (Ref.  1),  Haussmann  (Ref.  2),  Baras,  Elliott  and  Kohlmann  (Ref.  3) 
and  Elliott  (Ref.  4).  In  these  papers,  however,  the  control  variable  occurs  in  only  the 
drift  coefficient.  For  a  fully  observed  stochastic  control  problem  Bensoussan  (Ref.  5)  does 
consider  the  case  when  the  control  also  appears  in  the  diffusion  coefficient.  This  case  is 
also  discussed  in  (Ref.  6),  and,  when  the  optimal  control  is  Markov,  an  explicit  equation 
for  the  adjoint  process  is  derived. 

In  this  paper  we  consider  a  state  process,  which  is  only  partially  observed  through  a 
noisy  observation  process,  and  for  which  the  control  variable  is  present  in  both  the  drift 
and  diffusion  coefficients.  By  adapting  the  techniques  of  Bensoussan  (Ref.  5)  an  adjoint 
process  is  described  and  a  minimum  principle  obtained  for  an  optimum  control.  To  the 
best  of  our  knowledge,  this  is  the  first  paper  that  discusses  this  problem  for  the  partially 
observed  case  when  the  control  appears  in  both  the  drift  and  diffusion  terms. 

2.  Dynamics 

Suppose  that  the  state  of  the  system  is  described  by  a  stochastic  differential  equation, 
dxt  =  f(t,xt,u)dt  +  g(tyxt,u)dwt ,  xt  G  Rd ,  ro  =  0  <t  <T.  (1) 

The  control  parameter  u  will  take  values  in  a  compact,  convex  subset  U  of  some 
Euclidean  space  Rk. 

We  shall  assume  the  following: 
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(Al)  xo  €  Rd  is  given. 

(A2)  /  :  [0,  T]  xRd  xU  ->  Rd  is  continuous,  and  continuously  differentiable  with 
respect  to  x,u . 

(A3)  g:[0,T]xR.dxU-+Rd®Rn  is  a  continuous,  matrix  valued  function,  which  is 
continuously  differentiable  with  respect  to  x,u.  The  columns  of  g  will  be  denoted 
by  g W  for  k  =  1, 2, . . . ,  n. 

(A4)  There  is  a  constant  K  such  that 

(1  +  |x|)“1|/(t,a:,u)|  +  \fx(t,x,u)\  +  |/„(i,«,ti)|  <  K 
|0(<,*,«)l  +  +  |flr«(t,*,u)|  <  K. 

Suppose  the  observation  process  is  given  by 

dyt  =  h(xt)dt  +  dvt,  yt€Rm,  y0  =  0,  0  <  t  <  T.  (2) 

In  the  above  equations  w  =  ( w 1 , . . . ,  wn )  and  v  =  (u1 , . . . ,  vm)  are  independent  Brow¬ 
nian  motions.  We  also  assume: 

(A5)  h  :  Rd  — >  Rm  is  Borel  measurable,  continuously  differentiable  in  x,  and  for  some 
constant  K\ , 

\h(x)\  +  \hx(x)\  <  K\. 

Let  P  denote  Wiener  measure  on  C([0,T],f?n)  and  /x  denote  Wiener  measure  on 
C([0,T],  Rm).  Consider  the  space  Q,  =  C([Q,T),Rn)  x  C([0,T],Hm)  with  coordinate  func¬ 
tions  (t vt,yt)  and  define  Wiener  measure  P  on  Q,  by 


P(dw,dy )  =  P(dw)fi(dy). 


Definition  2.1.  Write  {Ft}  for  the  right  continuous,  complete  filtration  on  C([0,  T],i?n) 
generated  by  Ft°  =  cr{ wa,  s  <  t}.  Write  Y  =  {Yt}  for  the  right  continuous  complete 
filtration  on  C([0,  T],Rm)  generated  by  Yt°  =  cr{ys  —  yr,  0  <  r  <  s  <  t}.  The  set  of 
admissible  control  functions  ]7  will  be  the  Y-predictable  functions  on  [0,  T ]  X  C([0,  T],  Rm) 
with  values  in  U.  Then 


ILCL2Y{Q,T\ 

=  {t>(*X) :  v(t,w')  €  T2([0,T]  x  (C'([0,T],iem)),  dt  x  dp;  Rk ), 
for  a.e.  f,  v(t,-)  €  L2(C([OyT],Rm),Ytydp,Rk)}. 


For  u  €  U,  write  X“t(x)  for  the  unique  strong  solution  of  (1)  corresponding  to  con¬ 
trol  u,  and  with  X“s(x)  =  x. 

Write 

Z“,(*)  =  exp  (j‘h{X“r(x))'dVr  -  y‘\h{x;tr{x)) Is*)  (3) 

and  define  a  new  probability  measure  Pu  on  ft  by 


dPn 

dP 


—  ZqT{xq). 


Then  under  Pu,  ( Xo>t(xo)y  yt)  is  a  solution  of  (1)  and  (2). 

Cost.  We  shall  suppose  the  cost  is 

rp 

W+  /  t(r,XS,r(z„),ur)dr. 

Jo 

We  suppose 

(A6)  |C,(m){  +  \Cx{x)\  +  \Cxx(x)\  <  K(1  +  (m|g),  for  some  q  <  oo. 


3 


(A7)  t  .  [0,  '1  ]  X  R  x  U  *  R  is  Borel  measurable  and  continuously  differentiable 
m  ( x ,  u).  Furthermore  t  and  its  derivatives  in  x  and  u  satisfy  linear  growth 


conditions  in  x. 


The  expected  cost  if  a  control  u  €  U  is  used  is,  therefore, 

rjy 

J(u)  =  Eu  [C'(A'0u)r(a:o))  4-  J  £(r,XQ  r(xQ),ur)dr  . 


In  terms  of  P,  this  is 


T 

J(u)  -  E^ZqiT(xo)(c(Xq!T(xo))  +  J  d(r,Xott.(x 0),ur)drj|, 


Consider  the  d  +  1  dimensional  system  given  by 


Write 


-  x  +  £  f(rtXlr{x),Ur)dr  +  J  g(r,Xlr(x),ur)dWT 
Ki  =  *  +  £  ZlMXlr(x))dy  r. 


\  ( f<r>x:A*Ur) 

‘■'-[z *J  /W=1  0 


Hr)  = 


g(r,X»r(x),ur) 


Z«r(x,z)h(X?>r(x)) 


Wr  = 


Then  we  can  write  (4)  as 


=  *  +  jf  f(r,XZr(i),ur)dr  +  fg(r,Xlr(x),ur)dWr 
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dXu  AgN  _ 

As  in  (Ref.  3)  we  can  assume  the  Jacobian  — =  Z)“t  exists  for  all  s,  t,  x  and  all  w 
not  in  a  set  of  measure  zero,  and  is  the  solution  of 


~  f*  ~  ~  ~  n'*"m  ft 

Ds,t  =  I  +  /  9i\riXstr(x),ur)Ds>rdWr. 

Js  ;= 1  Ja 


(6) 


Here  I  is  (d  +  1)  x  (d  +  1)  identity  matrix.  In  fact,  if  the  coefficients  /  and  g  are  Ck  the 
map  x  — >  A“((5)  is  Cfc_1. 

Similarly  to  (Ref.  3)  the  matrix  process  H  defined  by 


~  ft  /  n+m 

Kl  =  I-  /  ff.*r(A(r,X,*.r(*).«r)-  £#(r,X»,(S), 

*=1  X 

m+n 

-  E  /  *,)<(»?. 


m+n 

E  / 

fc=l 

exists  and  tf  =  (5"  J"1 


(7) 


Remark  2.1.  Write  ||X“(x0)||,  =  sup  |X0“s(xo)|,  ||5uj|(  =  sup  |j?ffa|,  = 

°<*<l  0 <3<t  ' 

sup  Then  ||Au(x0)||r,  ||.DU||7’,  ||ifu||r  are  in  Lp,  1  <  P  <  oo. 

0  <s<t 

We  shall  suppose  there  is  an  optimal  control  u*  G  U,  so  that  J(u*)  <  J(u )  for  all 
other  u  G  U. 


Notation  2.1.  We  shall  write  X*  for  Xu *  and  L>Q  t  for  Dft,  etc. 


3.  Differentiability 

Suppose  u*  G  JZ  is  an  optimal  control.  Consider  any  other  control  v  G  £.  Then  for 
0G[O,1] 

ue{t)  =  u*(i)  +  6(v(t)  -  u*(t))  G  £ 

and 

J{ue)  >  J{u*).  (8) 
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If  the  Gateaux  derivative  J'(u*)  of  <7,  as  a  functional  on  the  Hilbert  space  iy[0,T], 
is  well  defined,  differentiating  (8)  in  6  implies 


{ J'(u*)>  v(t)  -  u*(t))  >  0 


for  all  v  G  JL- 


dX$  ,(x0) 

for  the  trajectory  associated  with  u*6.  Then  Mt  =  - ^ - 


LEMMA  3.1.  Suppose  v  €  H  is  such  that  u*e  =  u*  +  9v  6  U.  for  9  €  [0,  a].  Write  Xq  /^Xq) 

exists  a.s.  and  is  the 

“  WC7  |e_0 

unique  solution  of  the  equation 

Mt  =  /\/x(r,  X0r(5o)>  u*)Mr  +  /u(r,Xoy*0)XH)<*r 
Jo 

,=i  •'» 

+  E  /"j.  >(r. ^«V(*0),  (9) 


because  for  n  +  1  <  i  <  n  +  m,  <ju  ^  =  0. 

Proof.  The  result  follows  from  the  theorem  of  Blagovescenskii  and  Freidlin  (Refs.  7- 
8)  on  the  differentiability  of  solutions  of  stochastic  differential  equations  which  depend 
on  a  parameter.  In  effect  the  result  of  (Refs.  7-8)  states  that,  if  the  coefficients  are 
differentiable,  the  equation  for  the  derivative  is  obtained  by  differentiation.  Considering 
the  initial  condition  as  a  parameter  this  result  gives,  in  particular,  the  equation  for  the 
differential  or  Jacobian  as  in  (6).  □ 


Lemma  3.2.  Write 


rt  n  rl  ~  ~ 

—  It  ri*  \-i  (  j  /  m* 

,1  —  I  V^O ,rJ  Juy  J»r^'  I  I  V-^0 ,rJ  'Ju 

Jo  i=i  Jo 

-it,  /  (-^0 ,r)~19i){r)g^{r)vrdr 
i= i 


(10) 
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where  fu,  git  gu  are  as  in  equation  (9).  Then  Mt  =  Dq^qj. 

Proof.  By  differentiating,  we  see  the  product  DQ  tfj0>t  satisfies  equation  (9).  □ 

Lemma  3.3. 


d6 


jP 

+  /  fyr,XQir(xo),u*)DQrTjo)r  +  L{r,XQr(x0),u*)vr)dT 

J  0 


where 


c(x;iT(i„))  =  z;,T(z„)C(x;tT(x<,)) 
i(r,x;,r,K)  =  z;,r(*.)«(r,  xi  r(z„  ),<)• 


Proof. 

rp 

j{nl)  —  £■  +  Zo,t(xo)j  Kr’X*r{xo ),uj(r))dr)] 

=  S  p  A*  ,r(*o))  +  J„ZSAr'  <r(*o),  «;(r))<fr  +  J*  [j\(s)ds)dZ^ 

rp 

-  £j[c(Xq|T(xo))  +  i(r,Xoir(xo)yu*9(r))dr  . 


d0 


9=0 


=  £ 


rp 

ci(X!x(x0))MT+  j  (i£(r,x;,r(io),-4Wr 


+  l„(r,X0"  r(io),  u*)ur)drl 


substituting  Mt  =  DQ  tfj0lt  the  result  follows. 


□ 
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Consider  the  right  continuous  version  of  the  square  integrable  martingale 


Nt  =  E 


cs(l0-r(*o))50’,r+  f  > 'i(r,x;^.(io),K)S;,rdr  I  g, 

Jo 


where  Qt  is  the  right  continuous  complete  c-field  on  fl,  generated  by  Q\  —  F®  <g>  Y® . 
From  (Ref.  9)  Nt  has  a  martingale  representation 


tT  n+m  r*  — - 

wt  =  Bpi(x;,T(i0))S;iI.  +  jf  i(r,x;>r(i0),<*;)5;,r*]  +  E  JY'dW‘ 


where  the  7*  are  Qr  predictable  processes  such  that 


E 


<  00. 


Write 


it  =  JV,  -  f  Ur,Xlr(i»),<W,rdr 
Jo 


*  \-l 


ft  =  (tW,t) 


=  E 


Ci(x;tT(x0))DiT  +  / 4(r,  x0*  r(xo),  u;)5j,r*  1 5, 


Theorem  3.1. 


dJ(u|) 


d0 


9=0 


~E[Jo  PMs)v*ds  P*9i}(s)g^(s)v3ds  + 


(11) 
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Proof,  The  product  rule  gives 


fT  ~ 

Zt  ■  Vo ,T  =  /  fr(-Do,*)  1fu(s)vsds 
Jo 

+  E  [  UDl,r1g^(s)v,dWj  -  £  [T(,(Dl,)-'gP(s)gU(s)v,ds 
i=l  ^  i=l  Jo 

n+m  fT  __  yT 

+  E  /  -  /  h(.s)(Dl,)-%,,ds 

Jo  Jo 


+  £  [T(Dl,rlg^(s)v,j(‘>ds. 

i=i  Jo 


(12) 


However,  from  Lemma  3.3 


dJ(u^) 


dd 


rp 

?_0  =  -S  [&r  •  ^o.r  +  J  (h(s)DQt3ijo,s  +  44>a)dsj .  (13) 


Substituting  (12)  in  (13)  and  using  the  definition  of  p,  the  result  follows.  □ 

Remark  3.1.  Write  X*T(x)  =  (X*T(x),  Z*iT(x,  z))'  for  the  solution  of  (4)  using 
control  u*.  Then,  by  uniqueness, 


z;iT(x,z)  =  zzt*T(x,i) 


and  Z*tT(x,  1)  is  the  density  given  by  (3). 
Lemma  3.4. 


dZ*T(x,  z) 
dz 


ZIt(x, 1) 


—  Zlt  ^XO)  1)^0, t{xi  1) 


and 


dZlT(x,  1) 

dx 


Tdh{x*  ) 


dx 


•  D: 


t,rdvrj 


(14) 


(15) 

(16) 


(17) 
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where  D1  „  = 


*  —  . 

<>r  eta  ' 


Proof.  (15)  is  immediate  from  (14).  Now 


Z,>  {*,  i)  =  1  +  £  z;A*,  1)A  W»)A/r. 

Applying  the  differentiation  result  of  Blagovescenskii  and  Preidlin  (Ref.  7-8)  we  have 

ggfc(M)  _  [TdZtr(*,l  ).,y,  /%*  ,  *  a/i 

—  J  gx  KXttr(.x))dyr  +  J  ZtiJ.(x,l)  ^  [X tr{x))Dt)rdyr. 

This  equation  can  be  solved  by  variation  of  constants  to  give 

=  z«*,r(x,  -  [  fx(Kr(*))Kr  ■  M*«*,M)*) 

and  the  result  follows  from  (2).  L 

Notation  3.1.  Write  for  Zo)t(zo>  1),  for  Z*T{x,  1), 

«<)  =  (^(XoVM Kt  +  C«,T(*o))  (jf  Yx  •  D?.>-)  ’  Z»*T' '  W.r(*o))) 


*W  =  (4(r)i>r,r  +  <W (  J'  ~  ■  dv„) ,  Z„*7‘<!(r)) . 

Note  that  the  linear  growth  conditions  of  i  and  £x,  the  integrability  properties  of  D* 
and  the  boundedness  of  h  and  hx  imply  that 

J  {jt  'Kri)drl)dZi,r 


is  a  square  integrable  martingale. 
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Lemma  3.6. 


rp 

Pt  =  E *  [zj, ,  («i)  +  J  1>(r)dr )  |  ft] . 


(18) 


Proof. 

rp 

Pt  =  -E  ]Ci(X0*  7’(5o))Ht*7’ +  J  ^i(r,  -^O,r(io)i  Ur)D'  rd}‘  I  ft 

=  s[(z;,rC,(Jr0*T(*c)W,T  +  Zo/%— gw.rfa)).  t(*o))) 

+  [{ZlAWt,  +  Z«V  •  «(-•), Z.Vf’- 


Substituting  (17)  this  is 

=  E  Zo  T^Cx(Xq  T(xo))D*t  +  C(XlT{z0)(Jt  -q^  •  Dt.rdvr^j , Zo,tl C(X0iT(xo))} 

+ /\v{4(r)A% + «>•)(  [  §  •  ■  *,) .  i  s, 

T* 

=  e[z„VW  +  /  ZJ,^(r)dp|ft].  (W) 

Now 

T* 

e[/  Z0VV>(r)<lr  I  ft]  =  Z„”,e[j[  z,>(r)*|ft] 

=  Z;,,E[z('r/%(r)*- J  (jf  ^W^dZ.VIff,]. 

However,  the  last  term  is  a  square  integrable  (P,  £7j)  martingale,  so 

z;>r^(r)dr  I  &]  =  Z*0itE[z;tTJt  ip(r)dr  \  &] 

T* 

=  E[z;tTJ  tl>(r)dr  |  ft  . 


Substituting  in  (19) 


Pt  =  E 


z;,T(m + J  * (<•)*)  |  g, 


and  using  Bayes’  formula,  this  is 

=  E-  [«i)  +  J%(r)dr  |  gt]  Zl, 

fjt 

=  E'[zl,{m  + l  <Kr)dr)\  9, 


Definition  3.2.  The  adjoint  process  will  be  the  process  p  defined  by 

m,.  I  m 


m, ,  i  y.] 


□ 


p,  -  E\p.  |  Y.  V  {*}]  E[z^  !  n  y  {j)] 

-  +  I v  w]  ,  n  v  {.}] 

rp 

=  £*  [««)  +  J  ip(r )dv  |  n  V  |  Y,] 

J* 

=  E‘  [(«.)  +  J  Mr)dr)E{Zl,  |  r.)  |  Y.  V  {*}] . 

As  in  Bensoussan  (Ref.  1),  the  adjoint  process  depends  on  x,  which  represents  the 
state  of  the  process  at  time  s.  However,  x  is  just  a  parameter  which  is  integrated  out  in 
the  minimum  principle  of  Theorem  3.2. 

Returning  to  the  perturbation 


ug(t)  =  u*(t)  +  S(v(t)  -  u*(t )) 


of  the  optimal  control,  we  have 


d,J{ue ) 
d6 


6=  0 


>  0. 
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That  is 


EUo  A/“ww »-«*(*))*  -E/W’WSPWW*)  -«•(.))* 

yi 

+i  <*(a^)-“*W)*+E/^i(^.)-1#»(^)  -«•«)*]  > 


for  ail  u  £  17.  Now 


^[/  p*7*WW») -«*(«))*- f;  /  p40Mffli)M(v(s)  -  «*(<))* 

i'=l  •'o 

+ 1  e«(s)(v(s)  -  «•(.))*  +  g  (S)(„w  _  „«  w)*j 

= ^[/  *[*/.«  -  !>#«#(<) + 4W + Et;(S;, „)->?«(,) ,  K] 

•  (t/(a)  -  u*(i))da]  >  0. 


Therefore,  because  (20)  is  true  for  all  v  €  £,  we  have  for  a.e. 


t  and  a.s.  w. 


E[pJu(s)(v(s)  -  «*(«))  -  'jrrptgjp(s)gW(s)(v(3)  -  „*(,)) 

1=1 

+eu(sM,)  -  „*(4))  +  Eiri(5!,,)-,j<i>(<.X»W  -  «•(*))  j  n]  >  o 


for  all  v  6  U. 
Prom  (21) 


E[pJu(s)(y(s)  ~  u*(s ))  -  £p,#(s)#(s)(u(a)  - 

7^*  *  '  \  // 

«=l 

+4M«S)  -  u*W)  +  E7'(5;,.)->  j(0(j  )(„(i)  _  a.(<))  |  r<] 


13 


■  ww^1  WMii  towyiMW 


=  E 


b[zo*tWM+  /  <p(r )dr)  |  &]/,(.)(»(»)-«•())) 

-X>[zo*.r«M  +  I  Hr)dr)  S,M\s)g«>(s)(v(s) -«*(«)) 

1=1 

+  ZoA^tWMs)  -  «*-(«)) 

n 

+  z0>  £  z;,5.17i(5;i,)-I#(s)(„(s)  -  «•(.)) 

i=l 

T 

=  E*  [(<£($)  +  ^(r)drj  fn(s)(v(s)  -  u*(s)) 

"  /  /*r  \  ri 

“  E 1^(5)  +  /  ^(r)dr)9i  \s)9[!)(s)(v(s)  -  u*(s )) 

«=i  ^ 

+  ^0,T^(«X»W-«*W) 

+  E^r1^^)-1#^)^)  -  «*(*)) 

t=l 

T* 

=  E“  [(«»)  +  J  *(r)ir)E[Zl,  |  r.]/.(«)(»(«)  -  «*(»)) 

"  E  («'s)  +  f  <P(r)dr)E[Zl,  |  n]#Mj‘0«(t>«  -  «*(«)) 

1=1  '/a 

+  z;5«„M£[z;,,  |  k,](»(»)  -  «*(»)) 


yj  -e 
. 


z; 


+Ezj,r17;(B;„)-1s[z;,J  i  ly^ww*)  -  «*w)  i  n 


i=i 


>  0. 


Write 


h)  =  I Y.  v  {*}]  £[^[y'  =  E'[t(s)  I  n  V  |  rj 


-BK(c;,.)  1  I  y* v  Ml  £[Zj^  |’y,v{*}]' 


(22) 
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Define  the  Hamiltonian  by 


H(x,v,t,p(t ))  -  ptfu(t,x,v)  -  Y^Pt9i\t,x,u*)g(t,  x,v)  +  £(t,x,v)  +  V'-y 

>=1  «=i 

THEOREM  3.2.  Ifu*  is  the  optimal  control ,  then  a.e.  5 
[dH 

E  [  u  )5)P(<S))(U('S)  —  u*(s))  Tj  >  0  a.s. 


Proof.  Prom  (14),  fu(s)  and  ~g('\s)  (i  <  n)  are  F4  V  {x}  measurable.  Therefore, 

p 

0  <  ET [(#a)  +  jf  i,(r)ir)E[Zl,  |  r.)/„M(««  -  «*(<)) 

n  /T 

-  E  («*)  +  /  #0*)s[z;,.  I  -  «•(,)) 

i=l 

+  I  n](t>(»)  -  «*(»)) 

n 

+  E  I  nis^MMa)  -  «*(j))  y, 

»=1 

p 

=  s*{s*[(«a)  +  l  <Kr)ir)E[Zl,  I  rj/„(«Xi>M  -  «*(<)) 

n  *7* 

-£(#*)+  /  |n]#(5)^‘)(s)(u(5)-u*(5))  ■ 

t=l  •'J 

+  z;,r  4  (*w,.  I  r.lWa)  -  «*(.)) 

n 

+  Y,Zo7T^*0>s)-1E[Z*t3  I  Y,]gW(s)(v(s)  -  u*(s ))  |  Fa  V  {x}]  |  Y,} 

-  E*  [p»/tt(5)(w(«)  -  «*(«))  -  X)pa^,)(5)^tt)(5)(v(5)  -  “*(5)) 

:=1 

+  Eriztfti.)  I  y,  V  {*}]  •  E[ZI ,  I  y.](v(«)  -  u"(s)) 

n 

+  E  E'WJiWi,)-'  1  Y> v  M)*K,  I  rj^WWa)  -  “'(a))  r. 

i=l 
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E*  [p3/u(s)0>(s)  -  u*(s))  -  f2ps9ix)(s)g^(s)(v(s)  -  u*(s )) 

i= 1 

+  I y* v  W)  '  «*>  -  “*«> 

+  XJ-EKtSo',,)'1  I  y,  V  Ml  -gpr-ypr j|  •  Sl°  M(»M  -  «* W) 


=  E* 


Psfu(s)(v(s)  -  U*(s ))  -  Y^psgi\s)g{j\s){v(s)  -  U*(s)) 


«•=  1 


+  L(s)(v(s)  -  u*(s ))  +  ^ 73Pu }(s)(v(«)  -  «*(«))  I 

t=l 

=  E*  ^-(x,u*,s,p(s))(v(s)  -u*(s))  |  Y,]. 


So  the  result  follows. 


□ 
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Abstract:  A  finite  state  Markov  chain  is  considered.  Only  certain  of  its  jumps,  or 
alternatively,  only  the  total  number  of  its  jumps,  are  observed.  Based  on  this  information 
a  recursive  estimate  for  the  state  of  the  chain  is  derived.  The  novel  features  are  the 
representation  of  certain  basic  martingales  associated  with  the  Markov  chain,  and  the 
consequent  use  of  martingale  calculus  and  a  product  technique,  which  simplify  related 
formulae  and  calculations  in  the  book  of  Bremaud.  The  Zakai  equation  is  obtained  and  a 
related  control  problem  presented  in  separated  form. 


0.  Introduction 

A  finite  state,  continuous  time  Markov  chain  is  considered.  The  state  space  is  taken 

to  be,  without  loss  of  generality,  the  set  of  unit  vectors  S  =  {et- } ,  e*  =  (0, 0, . . . ,  1, . . . ,  0)* 

of  jR^+1,  thus  facilitating  the  use  of  linear  algebra.  Some  basic  martingales  associated 

with  the  chain  are  identified  and  natural  filtering  problems  discussed.  These  consider, 

for  example,  the  estimation  of  the  state  of  the  Markov  chain  if  only  the  total  number  of 

jumps,  or  the  number  of  jumps  into  certain  states,  are  observed.  Such  formulae  can  be 

obtained  by  specializing  a  general  result  in  the  book  of  Bremaud  [1];  however,  using  the 

Acknowledgments:  Research  partially  supported  by  NSERC  grant  A7964,  the  Air  Force 
Office  of  Scientific  Research,  United  States  Air  Force,  under  grant  AFOSR-86-0332,  and 
the  U.S.  Army  Research  Office  under  grant  DAAL03-87-0102. 
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basic  martingales  and  a  product  technique,  our  proofs  are  different  and  direct.  The  related 
Zakai  equation  for  the  unnormalized  conditional  distribution  is  then  obtained.  It  is  shown 
how  this  can  be  used  to  discuss  in  a  separated  form  the  associated  optimal  control  problem. 
Pull  details  of  the  latter  can  be  found  in  [3]. 


1.  Markov  Chain 

For  0  <  i  <  N  write  e,*  =  (0, . . . ,  1, . . . ,  0)*  for  the  i-th  unit  (column)  vector  in 
and  S  ~  {e{,  0  <  i  <  N}.  We  consider  a  finite  state  space,  continuous  time  Maxkov  chain 
{xt}>  i  >  0,  defined  on  a  probability  space  (0,j F,P);  without  loss  of  generality  the  state 
space  of  the  chain  is  taken  to  be  S.  Write  pj  =  =  e^).  We  suppose  for  some  family 

of  matrices  Af,  that  pi  =  (p® , . . . ,  pf  )*  satisfies  the  forward  Kolmogorov  equation 


=  AtPt- 


M  =  (ai j(t))y  t  >  0,  is,  therefore,  the  family  of  Q  matrices  of  the  process.  We  suppose 
K'jCOI  <  B  for  all  i,j  and  t  >  0.  Because  Af  is  a  Q-matrix 


«ti(0  =  “  ZJ  ajiW*  (1.2) 

Write  $(t,  s )  for  the  fundamental  transition  matrix  associated  with  A,  so,  with  I  the 
(JV  +  1)  x  (N  +  1)  identity  matrix 


-(<,  s)  =  At$(t,  s ),  $(s,  s)  =  I 


Li's: 

-r{t,  s )  =  -$(*,  s)A3,  $(t,  t)  =  I. 


Suppose  {Ft}  is  the  right  continuous,  complete  filtration  generated  by  X.  Then  for 


0  <  s  <  t,  if  Xs  =  x  <E  S, 


E[Xt  |  Fs]  =  Es,x[Xt ] 


=  $(f,s)z. 

LEMMA  1.1.  Mf.  } Q  -  Xq  -  /  ArXr-dr  is  an  martingale. 

JO 

Proof.  Suppose  0  <  s  <  t.  Then 

E[Mt  -  Ms  |  Fs]  =  E  \xt  -Xs-  f  ArXr-.dr  \  Fs 

=  E\xt-Xs-  C ArXrdr  |  Xs  , 

J  3 

(because  Xr  =  Xr-  for  each  u>,  except  for  countably  many  r), 

=  $(<,  s)Xs  -Xs-  ^ArHr,  s)Xsdr 
Js 

=  0  by  (1.3). 

COROLLARY  1.2.  By  variation  of  constants 

Xt  =  $(*,  0)(X0  +  J*$(r,d)~ldMr). 

NOTATION  1.3.  If  X  =  (a:o,^i,...,a;^)*  €  R ^r+1,  diag  X  will  be  the  diagonal  matrix 
with  entries  from  x.  For  x,y  6  write  x  ■  y  =  x*y  for  their  scalar  (inner)  product. 

Consider  0  <  i,j  <  N  and  i  ^  j.  Then 

{X3-  ■  ei)e*-dX3  =  {X3-  •  e;)e|  A Xs 


^iXs-^itfiXs-Xs-) 

—  I(Xs —  “  &i >  Xs  —■  Gj). 


3 


Define  the  martingale 


MtJ  :=  j\xs,  ■  ei)e)dMa. 


(Note  the  integrand  is  predictable.)  Then 

M\j  =  J‘(XS-  ■  e; )e)dXa  -  j'(Xa.  ■  e^A.X^ds. 

Wr'ting  Nt(i,j)  for  the  number  of  jumps  of  the  process  X  from  e*  to  e»  up  to  time  t  this 


—  -Nf(bi)  J  -f(-^Cs—  —  &i)aji{s)ds 
=  ~  J  I(XS  =  ei)aji(s)ds 


because  Xs  =  Xs-  for  each  w,  except  for  countably  many  s.  Therefore,  for  i  ^  j 


Nt(i,j)  =  f  I(XS  =  e;)a ji(s)d$  +  M?. 
J  0 


For  a  fixed  j,  0  <  j  <  N,  write  Nt(j)  for  the  number  of  jumps  into  state  e?-  up  to 


time  t.  Then 


mi)  =  'Em,}) = E  + m{ 

i  • _ i  JO 


7-i  ..  JO 

1  =  1  1  =  1 


where 


Finally,  write  JV^  for  the  total  number  of  jumps  (of  any  kind)  of  the  process  X  up  to  time  t. 


Then 


N  N  rt 


JV,  =  E  *<0')  =  E  %jAx‘=  ei>jMds + 9( 


j=l  i=lj=l 

¥j 


4 


where  Qt  is  the  martingale 


However,  from  (1.2) 


so 


N 

aii(s )  =  ~Yj  aji(s) 


&i)Q‘ii(s')ds  d*  Qt’ 


(1.4) 


2.  Filtering 

We  now  consider  the  recursive  estimation  of  the  state  Xt  given  the  number  of  jumps 
which  have  occured  to  time  t.  Other  counting  processes,  such  as  Nt(i,j),  Nt(j)  could  be 
considered  as  the  observation  process;  for  details  see  [3]. 

Notation  2.1.  Write 

N 

h(s,x3)  =  -£i(xs  =  ejte-iW 

i= 1 

and  a(s )  for  the  vector  (-aoo(5)>  •  •  • » ~aNN(s))*-  Then  h(s,Xs)  =  a(s)  •  Xs.  We  shall 
fuiiher  abbreviate  h(s,X3 )  as  h(s). 

We  have,  therefore,  a  SIGNAL  process 

Xt  =  X0  +  AsXs-ds  +  Ms  (2.1) 

J  0 

and  an  OBSERVATION  process 

Nt  =  t  h(s)ds  +  Qt-  (2.2) 

JO 
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Write  {If}  for  the  right  continuous  complete  filtration  generated  by  N,  so  Yf  C  Ft 
for  all  t.  If  {^f},  t  >  0,  is  any  process  write  for  the  ^-optional  projection  of  <f>. 
Then  <j>t  =  F[<f>t  \  If]  a.s.  Similarly,  write  <j>  for  the  y-predictable  projection  of  <f>.  Then 
<j>  =  E[(j>t  |  Yt-)  a.s.  From  Theorem  6.48  of  [2],  for  almost  all  a >,  <^f  =  cj>t  except  for 
countably  many  values  of  t.  Therefore 

f  h(r,Xr)dr=  f  h(r,Xr)dr 

Jo  Jo 

rt 

=  /  h(r,Xr-)dr. 

Jo 

Write  pt  =  Xt  =  E[Xt  |  Yt]  so  pg  =  #[Xg]  =  P0  say.  Now  h(r )  =  a(r)  ■  Xr  so  h(r)  = 
a(r)‘pr. 

For  the  vector  h(r)Xr  =  diag  a(r)-Xr  we  have  h(r)Xr  =  diag  a(r)-pr.  The  innovation 
process  associated  with  the  observations  is 

Qt  :=  jV}  —  f  h(r)dr 

Jo 

=  Nf  —  [  h(r—)dr. 

Jo 

Application  of  Fubini’s  theorem  shows  that  Q  is  a  {Yf }  martingale.  Therefore, 


rtn 

=  /  h(r—)d: 

Jo 


■r  +  Qt- 


Similarly,  Fubini’s  theorem  shows  that  the  process 


Mt  ’■=  Pt~P0~  [  AsPs-ds 

Jo 


is  a  square  integrable  fY*  I-martimrale.  Conseouentlv.  M  can  be  reoresented  as  a  stochastic 

A  O  V  V  J  "U  '  J>  V  t  *■ 


integral 


Mt=  f  IrdQr 
Jo 
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Therefore, 


E[Xt  |  Yi]  =  Pt  =  PQ  +  f  Arpr-dr  +  f  7 rdQr- 

JO  Jo 


The  problem  now  is  to  find  an  explicit  from  for  7. 


Theorem  2.2. 


7 r  =  I(p(r- )  ■  a(r )  ^  0)(p(r-)  •  a(r ))  1{diag  a(r)  ■  p(r~ ) 


-  (p(r-)  ■  a(r))p(r~)  +  Arp(r-)}. 


(2.4) 


Proof.  The  product  ptNt  is  calculated  two  ways.  First  consider 

XtNt=  Xr—(dQr  +  h(i  )dr) 

Jo 

+  f  Nr-.(ArXr-dr  +  dMr )  +  [X,  N]t. 

Jo 

Now  X  and  N  jump  at  the  same  times,  at  which  A N  =  1,  so 

[X,N]t=  &XrANr=  Y  A Xr  =  Xi-X0 

0  <r<t  0  <r<t 


=  /' 
Jo 


ArXr~dr  -j-  Mi. 


That 


is,  (X,N)t  =  f 

Jo 


ArXr-dr  so 


XtNt  =  f\xr-h(r~)  +  Nr-ArXr-  +  ArXr-)dr  + 
Jo 


m 


where  p  is  an  {Ft}  martingale. 

Taking  the  ^-optional  projection  of  each  side  of  (2.5) 


PtNt 


=  /  (diag  a(r)  •  pr_  +  Nr-Arpr-  +  Arpr~)dr  +  H} 

Jo 


(2.5) 


(2.6) 
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where  iJ1  is  a  square-integrable  Y  martingale.  However,  from  (2.3)  and  (2.4) 


ptNt  =  /  pr~h(i — )dr  +  /  pr-dQr  +  /  Arpr-Nr-dr 

Jo  Jo  Jo 


+  /  'frNr—  dQr  +  [p> -N]^. 

Jo 


Now 


[p,  Apr ANT  =  ]P  7rdNr 

0<r<<  0 <r<t 

=  /  yrdNr  —  /  JrdQr  +  /  7 rh(r—)dr. 

Jo  Jo  Jo 


Therefore, 

p<N<  =  /  ( pr-h(r -)  +  Arpr-Nr-  +  jrh{r-))dr  +  H f,  (2.7) 

*/0 

where  tf2  is  a  square-integrable  F  martingale.  The  bounded  variation  process  in  (2.6)  and 


(2.7)  must  be  equal,  so 


diag  a(r)  •  pr_  -f  Nr-Arpr _  +  Arpr_ 


=  prJh{r~)  +  Arpr-Nr..  A  jrh(r-). 


Recalling  h(j — )  =  a(r)  •  pr_  we  have 

7 r  =  Va#0)(a(r)'P(r~))“1{diaS  a(r)  ’Pr- 


-  ( a(r )  •  p(r— ))p(r— )  +  ^rp(r")}-  (2-8) 


Note  for  any  set  B  EYs 


E 
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so  7r  can.  be  taken  to  be  0  on  any  set  where  h(r— )  =  0. 

REMARKS  2.3.  We  have,  therefore,  that  pt  =  E[Xt  |  Yt]  is  given  by  the  equation 

Pt  =  PO  +  /  Arpr-dr  +  /  yr(dNr  -  a(r)  •  pr_dr) 

JO  J 0 

where  7r  is  given  by  (2.8).  The  disadvantage  of  this  equation  is  that  7  involves  the  inverse 
factor  (a(r)  •  pr-)~^. 


3.  The  Zakai  Equation 

Suppose  there  is  a  constant  k  >  0  such  that  —  at-t-(r)  >  k  for  all  i  and  r  >  0.  Then 
h(r)~l  =  (a(r)  •  Xr)-1  <  for  all  r  >  0.  Define  the  martingale  A  by 


A<  =  1  +  /  A r_(h(r— )  1  -  1  )dQT 
JO 


(3.1) 


and  introduce  a  new  probabihty  measure  P\  on  (f2,  F)  by 


E 


dP\ 

idP 


Ff]  =  A,. 


Then  it  can  be  shown  that  under  P\  the  process  Nf  is  a  standard  Poisson  process,  and  in 
particular  Qt  =  Nt  —  t  is  a  martingale.  Conversely  we  can  define  the  ( P\ ,  F)  martingale 

At  =  1  +  /*Ar_(h(r-)  -  l)dQr.  (3.2) 

Jo 

Then  Af  A^  =  1.  To  obtain  the  Zakai  equation  we  take  P\  as  the  reference  probability 
measure  and  compute  expectations  under  P\.  Write  IT(A^)  for  the  T-optional  projection 
of  A  under  P\.  Then  for  each  t  >  0,  II( Af )  =  £q[A(  |  Yt]  a.s.  It  can  be  shown  that 
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I 

n(A^)  =  1  +  /  A rdQr  where  Ar  =  II(Ar_)(h(r— )  —  1).  By  Baye’s  rule,  for  any  Ff 

J  0 

measurable  random  variable  <f) 

k  =  E[<P  I  Yt]  =  Eifci  |  YtyBrfKt  |  Yt]. 


Write  a(<j>)t  =  Ei[tit4>  I  Yt).  Then  a{Xt)  =  E^Xt  |  Yt)  =  qu  say,  and  a(  1)  =  II(At). 
Now  qt  is  an  unnormalized  conditional  distribution  of  Xt  given  Yt,  because  pt  =  E[Xt  | 
Yt]  =  9t/n(A<).  Calculating  the  product  II(A t)f>t  we  obtain  the  Zakai  equation  for  qf 


j  qt  =  PO  + 

i 

i 

i 

I  + 

I 

I 

|  This  equation  is  linear  in  q  and  the  inverse  (a  •  p)-1  has  disappeared. 


4.  Optimal  Control 

The  optimal  control  of  a  Markov  chain  when,  say,  only  the  total  number  of  jumps  is 
observed,  can  be  discussed  using  the  Zakai  equation  (3.3).  We  see  below  that  this  presents 
the  problem  in  a  separated  form.  We  suppose  the  family  of  Q- matrix  generators  Af(u) 
depend  on  a  control  parameter  u  €  U  (a  compact,  convex  subset  of  some  R^).  Write 


a(s,u)  =  ( -aoo(s,u),...,aNN(s,u ))* 


and  h(s,u)  =  a(s,u )  •  Xf  where  the  state  process  Xu  is  now  described  by  dynamics 


X?  =X%+  t Ar(u)X?_dr  +  Mf 
JO 
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for  0  <  t  <  T.  The  set  H  of  admissible  controls  is  the  set  of  {F*}-predictable  processes 
with  values  in  U. 

Write  Pi  for  the  probability  measure  under  which  Nf  is  a  standard  Poisson  process. 
Then  Qt  =  Nt  —  t  is  a  martingale  under  P\ .  For  each  u  €  U.  we  define 


ft  _ 

A^  =  1  +  /  7^_(h(r-,u)  -  1  )dQr. 

J  0 


n(Af )  is  the  F -optional  projection  of  A  under  P\  and  with 


*W)  =  «(«)«  \Yt) 


we  have  as  in  Section  3  that  the  unnormalized  distribution  satisfies: 


qt(u)  =  II(A Tt)pt(u). 


Write  Bi(u)  =  (diag  a(t,  u)—I+Af(u)).  Then  for  each  u  €  LL  the  unnormalized  distribution 
is  given  by  the  Zakai  equation 

9t(u)  =  P0+  /  Ar{u)qr-(u)du  +  f  Br{u)qr-(u)dQr. 

JO  JO 


COST:  A  function  on  the  state  space  S  is  represented  by  a  vector 


£  =  (£q,...,£n)*  e  RN+1. 


We  consider  for  simplicity  iust  a  terminal  cost  so  the  control  problem  is  that  of  choosing 


u  <=TJso  the  expected  cost 


J(u)  =  E{(e,x%) ] 


ii 


is  minimized.  Now 


J(u)  =  E^e.xf)] 

=  TftXft) 


=  B1[(e,E1[7^x^  |  y-,])] 

=  Me,  4)1 

The  control  problem  has,  therefore,  been  formulated  in  separated  form:  find  u  6  LL  which 
minimizes 

J(u)  =  £,[<<!,<#>] 

where  for  0  <  t  <  T,  qf  is  given  by 

3t(u)  =  PO  +  /  Ar(u)qr-(u)dr  +  [  Br(u)qr-(u)dQr. 

J  0  J  0 

Under  Pj,  Qt  =  Nt  -t  is  a  (Pi,  F)  martingale.  A  novel  feature  of  this  partially  observed 
control  problem  is  that  there  is  correlation  between  the  state  and  observation  processes 
which  leads  to  the  presence  of  the  control  u  in  the  “diffusion”  coefficient  B. 

A  minimum  principle,  and  an  equation  for  the  adjoint  process,  for  this  problem  are 
derived  in  [3]. 
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Abstract.  A  control  problem  is  considered  where  the  coefficients  of  the  linear  dynamics 
are  functions  of  a  noisily  observed  Markov  chain.  The  approximation  introduced  is  to 
consider  these  coefficients  as  functions  of  the  filtered  estimate  of  the  state  of  the  chain; 
this  gives  rise  to  a  finite  dimensional  conditional  Kalman  filter.  A  minimum  principle  and 
a  new  equation  for  an  adjoint  process  are  obtained. 

Key  Words.  Hybrid  control,  filtering,  minimum  principle,  adjoint  process,  separation 


principle. 


1.  INTRODUCTION. 


The  filtering  problem,  where  the  state  and  observation  processes  are  linear  equations 
with  Gaussian  noise,  has  as  its  solution  the  celebrated  result  of  Kalman.  For  the  related 
partially  observed,  linear  quadratic  control  problem  the  separation  principle  applies,  and 
the  optimal  control  can  be  described  explicitly  as  a  function  of  the  filtered  state  estimate. 

Suppose,  however,  the  coefficients  in  the  linear  dynamics  of  the  state  process  are 
functions  of  a  noisily  observed  Markov  chain.  Both  the  filtering  problem,  and  related 
quadratic  control  problem,  are  now  nonlinear,  and  explicit  solutions  are  either  difficult 
to  find  or  of  little  practical  use.  The  approximation  proposed  below  is  to  consider  the 
coefficients  in  the  linear  dynamics  to  be  functions  of  the  filtered  estimate  of  the  Markov 
chain.  In  this  way  a  conditional  Kalman  filter  can  be  written  down.  These  dynamics  lead 
us  to  consid'  r  in  Section  3  a  conditionally  linear,  Gaussian  control  problem.  By  adapting 
techniques  of  Bensoussan,  Ref.  1,  a  minimum  principle  and  a  new  equation  for  the  adjoint 
process  are  obtained. 

Other  work  discussing  similar  situations  and  approximations  includes  the  papers, 
Refs.  2-6  and  the  recent  book,  Ref.  7  by  Mariton. 


2.  DYNAMICS. 

Consider  a  system  whose  state  is  described  by  two  quantities,  a  vector  x  €  Rd  and  a 
component  a  which  can  take  a  finite  number  of  values  from  a  set  S  —  {ui,  {?2, . . . ,  <Jn}- 
(x  can  be  thought  of  as  describing  the  location,  velocity  etc.,  of  an  object;  a  might  then 
describe  its  orientation  or  some  other  operating  characteristic.) 
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Let  <f>i  be  the  function  on  S  described  by 


f  1  if  <7  =  <7,- 

<f>i{a)  =  l 

(  0  if  v  Oi 

and  write  (f>(a)  for  the  column  vector  .  • . ,  <t>  is,  therefore,  a  bijection  onto 

the  set  of  unit  column  vectors  {ei , . . . ,  e#}  of  RN ,  where  e,-  =  (0, . . . ,  0, 1, 0, . . . ,  0)'. 

If  a  evolves  as  a  Markov  process  on  S  we  can,  without  loss  of  generality,  consider  the 
corresponding  process  described  by  ^  evolving  on  the  set  {ei,...,ew}.  Write  <f>t  for  the 
state  of  this  process  at  time  t  and  pt  =  E[<f>t].  Suppose  the  generator  of  the  Markov  chain 
is  the  Q  matrix  Q(t)  -  ( qij(t )),  1  <  ij  <  iV,  so  that  pt  satisfies  the  forward  equation 

Tt  “  f1) 

It  follows  from  (1)  that  on  the  family  of  er-fields  generated  by  <pt  the  process  Mt  is  a 
martingale,  where 

Mt  =  4>t  ~  <f>o  ~  /  Q(s)<f>sds.  (2) 

Jo 

Suppose  <{>  is  observed  only  through  the  noisy  process  z,  where 

zt=  [  r(s,  <j>3)ds  +  Vf  (3) 

Jo 

Here  u  is  a  Brownian  motion  independent  of  M.  Write  {Zt}  for  the  right  continuous 
complete  family  of  cr-fields  generated  by  2  and  4>t  for  the  Z-optional  projection  of  <j>,  so 
that 

6*  =  E\(b *  I  Z*  1  a.s, 

*  *  i#  *  i  - J 

(For  a  discussion  of  optional  projections  see  Elliott  [4].  Optional  projections  take  care  of 
measurability  in  both  t  and  w\  conditional  expectations  only  concern  measurability  in  w.) 
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Write  A(s)  for  the  vector  (r(s,ei), . . .  ,r(.s,  e#))  and  diag  A(s)  for  the  diagonal  matrix 
with  diagonal  A(s). 

With  an  innovation  process  t>t  given  by  dut  =  dzt  —  (A(f),  (j>t)dt  it  is  shown  in,  for 
example  Ref.  8,  that  the  equation  for  the  filtered  estimate  (j>  is 

=  o+  [  Q(s)(j>sds  +  [  (diag  A(s)  —  (A(s),<j>s)I)<psdus.  (4) 

Jo  Jo 

Here  (  ,  )  denotes  the  inner  product  in  RN’  and  I  is  the  N  >.  N  identity  matrix.  Equation 
(4)  provides  a  recursive  expression  for  the  best  least  squares  estimate  <j>  of  <f>  given  the 
observations  z. 

Suppose  now  the  x  component  of  the  state  is  described  by  the  equation 

dxt  =  A{(f>t)xtdt  +  ptd<f>t  +  B((pt)dwt.  (5) 

Here  x  €  Rd,  wt  —  (wj , . . .  ,w")  is  an  n-dimensional  Brownian  motion  independent  of  M 
and  v,  and  A(<f>t),  B(<f>t)  and  pt  are,  respectively,  d  X  d,  dxn  and  dx  N  matrices.  Note 
that 

N 

i=i 

N 

*(*<)“  £>(*)<«>*)• 

1  =  1 

Suppose  the  x  process  is  observed  through  the  observations  of  y,  where 

dyt  =  Hxtdt  +  Gdjdt •  (6) 

Here  y  €  Rp,  fit  —  (/?*, . . . ,  /2(m)  is  an  m-dimer  signal  Brownian  motion  independent  of  M, 
v  and  w  and  if,  (resp.  G ),  is  a  p  x  d  (r-esp.  "  ,u  singular  p  xm)  matrix. 
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Now  the  y  observations  also  provide  information  about  <f>,  so  that  altogether  we  have 


with  observations  given  by  (3)  and  (6).  Write  {K<}  for  the  right  continuous,  complete 
filtration  generated  by  y  and  z,  and  denote  by  a  bar  the  F-optional  projection  of  a  process 
so  that,  for  example, 

],t  =  E{<i>t\Yt}  a.s. 


Define  the  if -innovation  processes  t/*,  0*  by 


du *  =  dzt  —  (A (f),  (f>t)dt 

d0*  =  G~1(dyt  -  Hxtdt). 


This  is  a  nonlinear  equation.  However,  the  approximation  we  shall  make  is  to  suppose 
that  most  of  our  information  about  (f>  comes  from  the  observations  of  2  and  that  we  can 

A  At  A, 

replace  <j>  by  <f>  in  (5),  where  <f>  is  given  by  (4).  Note  that  <f>  is  independent  of  w  and  0.  We 
can,  therefore,  sta*e  the  following  result: 


PROPOSITION  2.1.  Suppose  the  state  xt  is  approximated  by  xt  where 

dxt  =  A(<f>t)xtdt  +  ptd<j>t  +  B((j>t)dvjt.  (8) 

Here  <b  is  given  by  (4).  Suppose  x  is  observed  through  the  process  y  where 

dyt  =  Hxtdt  +  Gdflt-  (9) 

Write  {Yi}  for  the  right  continuous,  complete  filtration  generated  by  y  and  x  for  the 
Y -optional  projection  of  x,  so  that  xt  =  E[xt  |  Yt]  a ,s.  Then 

dxt  =  A(it)xtdt  +  ptd$t  +  PtH(GG')-ldpt  (10) 

x0  =  Ex0, 

where 

G  -d/3t  =  dyt- Hxtdt  (11) 

and  Pt  is  the  matrix  solution  of  the  Riccati  equation 

Pt  =  -  PtH\GG')~XHPt  +  A(<j>t)Pt  +  PtA{h\  (12) 

Po  =  cov  Xq. 


Proof.  Because  <f>  is  independent  of  w  and  B,  appears  as  a  parameter  in  (10), 


so  the  usual  Kalman  niter  formula  applies.  E 


quations  (4),  (10),  (11)  and  (12)  therefore, 


give  a  finite  dimensional  filter  for  xt,  which  is  a  conditionally  Gaussian  random  variable 
given  (f>  and  Yt. 
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Note  that 


N 

^  A(ej){e j,  <j>t) 

:= 1 

N 

1=1 

Remarks  2.3. 

d(xt  -  xt )  =  ( A(4>t)xt  —  A(4>t)xt)dt 

+  PtQt(j>t  -  4>t)dt  +  (2  ®  Hx  -  x  <g>  Hx)G~ld(3 * 
+  (a;  <2>  (A  (t),  <f>t)  -x®  (A  (t),  4> t))du * 

+  PtH(GG')~1dpf 

Therefore,  with  tr  denoting  the  trace  of  a  matrix , 
d(xt  -  xt)2  =  2(xt  -  2 t)d(xt  —  xt) 


□ 


+  tr(x  ®  lix  —  2  ®  Hx)(G'G)  1  [Hx  ®  2  —  Hx  <g>  x)dt 
+  tr((  A(t),  (j>)®x-  (A  (f),  <£)  ®  2)  (2  ®  (A(i),  <f>)  -  x  ®  (A  (t),  <£))<& 

+  trPtH( GG' )~2H'P't  ■  dt 
+  trPtH(GG')~1  •  G'(x  ®  Hx  —  2  ®  Hx)dt. 

Taking  expectations  the  martingale  terms  disappear  and,  under  integrability  or  bounded¬ 
ness  conditions  on  the  coefficient  matrices,  an  estimate  of  order  o(t )  for  E(xt  —  it)2  can 
be  obtained.  However,  this  does  not  appear  too  useful. 


8 


3.  HYBRID  CONTROL. 


Suppose  the  state  equation  for  x  now  contains  a  control  term,  so  that 

dxt  =  A(<f>t)xtdt  +  ptd(f>t  +  Ctu(t)dt  +  B((f>t)dwt.  (13) 

The  observation  process  is  again  y,  where 

dyt  =  Hxtdt  +  Gdf3t.  (14) 

Assume  the  control  parameter  u  takes  values  in  some  space  Rk  and  the  admissible  control 
functions  are  those  which  are  predictable  with  respect  to  the  right  continuous,  complete 
filtration  generated  by  y  and  (f>.  C*  is  a  d  x  k  matrix. 

Suppose  the  control  {u*}  is  to  be  chosen  to  minimize  the  cost 

J(u)  =  Eyj  ( x'tDtxt  +  u'iRtu^dt  +  .  (15) 

Here  Dt,  Rt  and  F  are  matrices  of  appropriate  dimensions  and  Rt  is  non-singular.  Then 
(7),  (3),  (13),  (14)  and  (15)  describe  a  nonlinear  partially  observed  stochastic  control 
problem  whose  solution  is  in  general  difficult.  To  obtain  a  related  completely  observed 

A 

problem  the  approximation  we  propose  is  that  <fit  is  replaced  by  its  filtered  estimate  <j>t  in 
(13)  giving  a  process  x,  where 

dxt  =  A(<j>t)xtdt  +  ptd(j>t  +  Ctu(t)dt  -{-  B(<f>t)dwt.  (16) 

The  observation  process  is  now  y,  where 

dyt  =  Hxtdt  +  G  •  df3t  (17) 
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and  the  admissible  controls  are  the  predictable  functions  with  respect  to  the  right  contin¬ 
uous,  complete  filtrations  generated  by  y  and  2. 

The  cost  is  taken  to  be 


J(u)  =  E  /  ( x[DtXt  +  u\RtUt)dt  +  x'tFxt 


(18) 


Equations  (16),  (17)  and  (18)  describe  a  partially  observed,  linear,  quadratic  Gaussian 
control  problem  which  is  parametrized  by  $*,  a  process  which  is  independent  of  w  and  jJ. 
However,  we  cannot  apply  the  separation  principle,  as  in  Ref.  9,  because  the  coefficients  in 
(16)  are  functions  of  <f>.  The  usual  form  of  the  separation  principle  involves  the  solution  of 
a  Riccati  equation  solved  backwards  from  the  final  time  T,  and  we  do  not  know  the  future 
values  of  <f>.  We,  therefore,  proceed  as  follows  to  derive  a  minimum  principle  satisfied  by 
an  optimal  control.  We  are  in  effect  considering  a  completely  observed  optimal  control 

A 

problem  with  state  variables  <f>  and  x,  where  <j>  is  given  by 


4>t  =  <£o  4-  /  Q(s)4>(s)ds  +  f  TL(s)(f>(s)dus 
Jo  Jo 


(19) 


and 


xt  =m0  + 


f  A(i,)x,ds+  f  p3d<j>a  +  /  C3usds+  [  P3H(GG'yldps.  (20) 
Jo  Jo  Jo  Jo 


Here  n(s)  =  diag  A(s)  —  (A (s),<j>3)I  and  mo  =  Ex0.  Note  from  (12)  that  the  co- 
variance  Pt  depends  on  <f>  In  terms  of  x  and  P  the  cost  corresponding  to  control  {u<}  is, 
(see  Ref.  9), 

*T 

J{u)  =  E  \^j  (x'tDtxt  +  u'tRtut)dt 

+  x'tFxt  +  J  tr(PtDt)dt  +  tr(PrF)J . 
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l.llUkli'U 


The  last  two  terms  do  not  depend  on  the  control,  so  we  shall  consider  a  problem  with 
dynamics  given  by  (19)  and  (20),  and  a  cost  corresponding  to  a  control  u  given  by 


J(u)  =  E^J  ( x'tDtxt  +  u'tRtu()dt  +  x'TFxq 


(21) 


Write  {Yt}  for  the  right  continuous  filtration  generated  by  y  and  2.  Write  Ty  [0,  T\  — 
{u(t,io)  6  L2([0,T]  x  0;  dt  x  dP,  Rk )  such  that  for  a.e.  f,  u(t,  •)  6  L2(Q,,Yt,P,Rk)}. 
Assume  U  is  a  compact,  convex  subset  of  Rk.  Then  the  set  of  admissible  controls  is  the 
set 

U_  =  {u  €  Ly[0,T] :  u(t,u>)  G  U  a.e.  a.s.}. 


Suppose  there  is  an  optimal  control  u*.  We  shall  consider  perturbations  of  u*  of  the  form 
ug(t)  =  u*{t)  9(v(t)  -  u*(t ))  where  v  is  any  other  admissible  control  and  9  £  [0,  lj.  Then 


J{ue)  > 


Following  and  simplifying  techniques  of  Bensoussan,  Ref.  1,  our  minimum  principle  is 
obtained  by  investigating  the  Gateaux  derivative  of  J  as  a  functional  on  the  Hilbert  space 
Zry[0,T].  Write  x*  for  the  trajectory  corresponding  to  the  optimal  u*.  Then 

dx\  =  A(<f>t)x*dt  +  ptdj>t  +  Ctu*dt  +  PtH(GG')~xdf3f 

Given  any  sample  path  will  be  considered  as  a  time  varying  parameter.  Write 

$(<|i,i f,s)  for  the  matrix  solution  of  the  equation 

^  $(<M,s)  =  A((f>t)$(<i>,t,s)dt 

with  initial  condition  $(<^,s,s)  =  I. 
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Lemma  3.1.  Suppose  v  €  LL  is  such  that  u*e  =  u*  +  8v  €  U_  for  9  G  [0,a].  Write  xe  for 

,t  I 


the  solution  of  (20)  associated  with  u*6.  Then  ipt  = 

ipt  =  $(<£,*,  0)  [  $(<j>,s,0)~1C3v3ds 
Jo 


exists  a.s.  and 


(22) 


Proof. 


*0|i  ~  "i~ 


f  A(j>3)xeads  +  /  M#.+  f  C»(u*  +  Ov3)ds  4-  fPsH(GG')-ldp3.  (23) 

JO  JO  Jo  Jo 


Prom  the  result  of  Blagovescenskii  and  Freidlin,  Ref.  10,  on  the  differentiability  of  solutions 
of  stochastic  differential  equations  with  respect  to  a  parameter  (23)  can  be  differentiated 
to  give 

*l>t  =  f  A(63)z3ds  +  [  C3v3ds.  (24) 

Jo  Jo 


The  solution  of  (24)  is  then  given  by  (22). 
NOTATION  3.2.  Consider  the  martingale 

rT 


□ 


Mt=E 


and  write 


2  /  x*'Ds§{4>,  s,  0 )ds  +  2 T,  0)  |  Ft 
■  Jo 

Zt  =  Ml-2[tx:'DMJ>,S,0)ds 

Jo 


pt  =  o) 


-1 


(25) 


Vo ,t=  f  $(<£,s,0)-1C>sd,3. 
Jo 


Then  there  are  square  integrable  processes  7  and  A  such  that  the  martingale  M  has  a 
representation  as  a  stochastic  integral 

Mt  =  E  [2  jJx*'D9${]>,  s ,  0)ds  +  2  x%F$(l  T,  0)]  +  j\3d$s  +  jf  \3du3. 
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Proposition  3.3. 


dJ{u*g) 

do 


6= o 


Proof. 


fT 

J(u*6)  =  E[jo  (xes'Daxe3  +  u*0'sRau*ds)ds  +  3%Fx^ . 
Therefore,  differentiating  we  see 
dJ ( ug )  I 


dd 


9=0 


=  E 


^  J  Dsza  +  u3  Rava)ds  +  2xj: Fzt 


Using  the  above  notation 


fa  =  $(<f>,t,0)T]o,t 
St  =  2x*1IF$(It,0) 


£tt]o}t  =  2  xj'FipT 


=  /  s,  0)  'Cavads-  2/  xl'D.Qd,  s,0)t}0ads 
Jo  Jo 

f 

Jo 


rT  „  rT 

+  /  7 sVo,ad0a  +  /  Xsijo,adva. 
Jo 


Substituting  in  (26) 


dJ{u*0) 

de 


T 

?=o  =  E  rr?0'T  +  2 1  0)i7ol4cfc  +  2J  u*'Ravads} 

-rr  — 

~  E\J0  5>  0)  l^-'svsds -\-2  J  u*a'Ravads 


=  E 


[  (PsCav 

Jo 


a  +  2  u*s'Rava)ds 


(26) 
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□ 


Now  take  v  to  be  of  the  form  v  —  u*  so  that  u$  =  u*  +  9(v  —  u *)  €  U,-  Applying 
proposition  3.3  to  J{ug )  we  have  the  following  result. 

COROLLARY  3.4.  The  optimal  control  satisfies  the  minimum  principle 


psCau*  +  2u*'Rsua  =  nun (p,Csv  +  2 u*s'Rav)  a.e.  a.s. 


Proof,  u*  is  optimal  so  |  >  0,  that  is  for  any  other  admissible  control  v 

E\ [T(paCa(u's-va)  +  2u*'Rs(U*3-v3))ds\  >0. 

LJ  o 

v  can  equal  u*  except  on  an  arbitrary  set  of  the  form  A  x  [s,s  +  h],  A  €  Fs.  Therefore, 


a.e.  dt  and  a.s.  dP, 


PsCa(u*a  -  va)  +  2 u*a'R3(u*s  -  va)  >  0, 


where  the  adjoint  variable  p  is  given  by  (25).  □ 

REMARKS  3.5.  Prom  Ref.  11  we  know  the  optimal  control  u*  is  feedback,  in  the  sense 
that  at  time  t  it  is  a  function  of  the  states  it  and  <f>t-  However,  to  avoid  derivatives  of  u* 
we  suppose  u*  always  follows  the  trajectories  of  x*  and  i,  even  if  these  trajectories  are 
perturbed.  By  the  Markov  property  we,  therefore,  have  that  pt  is  a  function  of  x  =  x t  and 
(f>  =  Writing  $  =  0)  and  y  =  2  /  x*'Da$(<j>,$,0)ds  we  have  that  z,y,  4>)  = 

Jo 

pt(x ,  <f)  •  $  +  y  =  Mt,  a  martingale. 

If  we  write  down  the  Ito  expansion  of  T  the  sum  of  the  terms  integrated  with  respect 
to  time  must  be  zero.  After  division  by  $  we  have  the  following  equation,  satisfied  by  the 
adjoint  process  p  =  p(t,  x,  <j>). 
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PROPOSITION  3.6.  Denote  the  Hessian  of  p  with  respect,  to  x  (resp.  <f>)  by  (resp. 


and  write 


Tt  =  ptH{GG')-1  +  ptn(t)$t 

A  t  = 

and  fr(F't  F<)  (resp.  tr(A'(  A*))  for  the  vector  with  components  tr(r't  P() 
(resp.  tr(A't  ~p  A,)). 

Then 

+  P 4  A(<j>t)  +  ^  4-  +  Ctu* ) 

+  TfQ*‘+1fr{r'  0r-)  +  l'r(AiSA,)  =  o' 

with  terminal  condition  p(T,  x,  f>)  =  2rF. 
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1.  Introduction 

A  finite  state  space  Markov  chain  is  considered.  Without  loss  of  generality  its  state 
space  can  be  taken  to  be  the  set  of  unit  basis  vectors  of  .  On  the  basis  of  knowing  only 
the  total,  number  of  jumps  a  control  problem  is  discussed  in  ‘separated’  form.  That  is, 
the  Zakai  equation  for  its  unnormalized  distribution  is  taken  as  describing  the  state  of  the 
process.  This  is  a  linear,  vector  equation  driven  by  a  standard  Poisson  process  in  which  the 
control  variable  also  appears  in  the  ‘diffusion’  coefficient  multiplying  the  noise  term.  The 
controls,  similar  to  those  employed  by  Bismut  [2]  and  Kushner  [4],  are  in  the  ‘stochastic 
open  loop’  form.  By  adapting  techniques  of  Bensoussan  [1]  and  calculating  a  Gateaux 
derivative,  the  minimum  principle  satisfied  by  an  optimal  control  is  obtained.  Finally, 
when  the  optimal  control  is  Markov,  the  integrand  in  the  martingale  representation  can 
be  obtained  explicitly,  and  new  forward  and  backward  equations  satisfied  by  the  adjoint 
process  derived.  A  full  treatment  can  be  found  in  [3]. 


2.  Dynamics 

Without  loss  of  generality,  the  state  space  of  a  finite  state  Markov  chain  can  be 
identified  with  the  set  S  =  {et-}  of  unit  vectors  ez-  =  (0, 0, . . . ,  1, . . . ,  0)*  of  .  Assume, 
therefore,  that  Xf,  t  >  0,  is  a  Markov  process  defined  on  a  probability  space  (fl,  F,  P) 
with  state  space  S  =  {ej, . . . ,  e^y}.  Write  p\  —  P(Xt  =  e*),  1  <  i  <  N,  and  suppose  for 

Acknowledgments:  Research  partially  supported  by  NSERC  grant  A7964,  the  Air  Force 
Office  of  Scientific  Research  United  States  Air  Force,  under  contract  AFOSR-86-0332,  and 
the  U.S.  Army  Research  Office  under  contract  DAAL03-87-0102. 
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some  family  of  matrices  At(u )  that  pt  =  satisfies  the  forward  Kolmogorov 

equation. 

^  =  At(u)pt . 

Here  £/,  the  set  of  control  values,  is  a  compact  con%rex  subset  of  some  Euclidean  space  . 
Take  0  <  t  <  T  and  suppose  At(u)  is  measurable  on  [0,T]  x  U  and  continuously  ‘fferen- 
tiable  in  u. 

The  set  £7  of  admissible  control  functions  is  the  set  of  U  valued  functions  which  depend 
only  on  the  knowledge  of  previous  jump  times.  That  is.  if  2\,T2, . . .  are  the  jump  times 
of  X  and  Tn  <  t  <  Tn+\,  then  u  £  U  is  a  function  only  if  2\, . . . ,  Tn  and  t.  It  is  easily 
checked  that  the  process  is  a  martingale,  where 

Mf  :=  X?  -X0  -  t A~(u)X?._dr, 

JO 

Write  Nf  for  the  total  number  of  jumps  to  time  t.  We  shall  suppose  our  only  knowledge 
of  the  Markov  chain  X  comes  from  observing  N.  Y  =  {¥(}  is  the  right  continuous, 
complete  filtration  generated  by  N.  For  u  €  U_  set 

a(s,  u)  -  (-an(s,  u), . . . ,  -aNN(s,  u ))* 
and  h(s,u )  =  a(s,u )  •  X% . 

The  hat^will  denote  the  K-optional  projection,  so  that  ps(u)  :=  Xg  •=  E[Xg  j  F3]  a.s, 
and  h($,u)  =  a(s,u)  ■  ps(u).  Now  N  can  be  written 


where  Qu  is  a  ( Y ,  P )  martingale. 

Note  that  because  Xu  and  N  jump  at  the  same  time  the  noises  in  the  state  process 
Xu  and  observation  process  N  are  correlated. 

Cost:  A  real  function  on  the  state  space  S  =  {ei,...,en}  is  given  by  a  vector 
l  ~  (£1, . . .  n).  Write  (£,  x)  =£*  -  x  for  the  inner  product  on  . 

The  control  problem  will  be  that  of  choosing  u  £  U_  so  the  expected  cost  J(u)  = 
E[{Z,X j.)]  is  minimized. 
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Suppose  there  is  a  k  >  0  such  that  —a^(s,u)  >  k  for  all  i,  all  s  £  [0,T]  and  all  u  €  U . 
Then  =  ( a(s,u )  ■  Xf)~^  <  k~ For  u  £U_  consider  the  martingale  Au  defined 

by  t 

A?  =  1  +  /  AJL {h(r-,u)~l  -  1  )dQ*. 

JO 

Then,  consider  the  new  measure  Pu  on  (fl, F)  given  by  E[^jj$-  |  Ft]  =  A]1.  It  can  then  be 
shown  that  under  Pu  the  process  Afy  is  a  standard  Poisson  process;  in  particular  Qi  =  Nt  —t 
is  a  martingale. 

The  inverse  of  A.f  is  the  process  A^,  a  (PU,F)  martingale  given  by  the  equation 

ft 

A?  =  1  +  /  A r-(h(s-,u)  -  l)dQ%. 

Jo 

Write  H(A?)  for  the  F-optional  projection  of  A^  under  measure  Pu  and  consider  the 
unnormalized  conditional  expectation  of  X  given  by  qt(u)  :=  Eu\XfXt  |  l*].  From  Bayes’ 
rule  qt(u )  =  II(Af  )pt(u),  and  with  Bf(ti)  =  (diag(a(t,u))  +  At{u)  —  I),  qt(u)  is  given  by 
the  Zakai  equation 

qt{u)  -PQ+  f  Ar{u)qr-{u)dr  +  /  Br{v)qr-(u)dQr. 

Jo  Jo 

Furthermore,  the  cost  can  be  written 

J(u)  =  Eu[(e,q%)).  (1) 

The  partially  observed  control  problem  has,  therefore,  been  formulated  in  separated  form: 
find  u  £JJ_  which  minimizes  J(u)  given  by  (1),  where  q  is  given  by  the  Zakai  equation  in 
which  Qt  =  Nt  —t  and  Nt  is  a  standard  Poisson  process. 


3.  Minimum  Principle 

Suppose,  therefore,  P\  is  a  probability  measure  under  which  N  is  a  standard  Poisson 
process,  and  for  u  6  ££  consider  qt(u)  €  defined  by 

?t(u)=P0  +  /  Ar{u)qr-(u)dr  +  /  Br{u)qr-{u)dQr. 

Jo  Jo 


The  cost  corresponding  to  u  €  U_  is  J(u)  =  Ei[(£,qj,}]. 

For  u  €  17  write  $w(t,  s)  for  the  solution  of  the  matrix  equation 

d$u(t,  s )  =  At(u)$u(t-,  s)dt  +  Bt(u)$u(t-,  s)(dNt  -  dt) 

with  initial  condition  $u(s,  s)  =  I. 

Furthermore,  for  u  €  iZ  consider  the  matrix  ^u(t,  s )  defined  by 

■$u{t,s)  =  I  -  ft^u(r-,s)Ar(u)dr  -  t  $u{r-,s)Br{u)dQr 

Jo  Jo 

ft 

+  /  *u(r-,s)BZ(u)(I  +  Br(u))-ldNr. 

Jo 

Then  it  is  easily  checked  that 

$u(t,s)yu(t,s)  =  I. 

Suppose  there  is  an  optimal  control  u*  €  J7.  Write  q*  for  qu  ,  $*  for  ,  etc.  Consider 
any  other  control  v  €  f£.  Then  for  0  €  [0, 1], 

u0{t)  =  u*(t)  +  0(v(t)  -  u(t))  e  U. 

Because  U  C  f?*"'  is  compact,  the  set  U_  of  admissible  controls  can  be  considered  as  a 
subset  of  the  Hilbert  space 

H  =  L2[Clx  [0,T] :  J2*J. 

Now 

JM  >  J(u*).  (2) 

Therefore,  if  the  Gateaux  derivative  J'(u *)  of  J,  as  a  functional  on  the  Hilbert  space  H, 
is  well  defined,  differentiating  (2)  in  0,  and  evaluating  at  0  =  0,  implies 

(/(«*),  »(0 -»*(*))  >o 


for  all  v  €  U_ . 
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LEMMA  3.1.  Suppose  v  €  U  is  such  that  Uq  =  u*-\-6v^.JJ_  for  9  €  [0,  a].  Write  qt(Q)  for 
the  solution  qt(ug)  of  the  Zakai  equation.  Then  zt  =  „  exists  said  is  the  unique 

solution  of  the  equation 


6=0 


zt  =  J  (^(r,u*)jvrq*_dr  4-  J  Ar(u*)zr-dr 


+ 


tt  .  ag  .  _  rt  _ 

JO  \^(r,u*'))Vrq*~dQr  +  jQ  Br{u)zr-dQr- 


(3) 


Proof. 


qt(Q)  =P0+  f  Ar(u*  +  6v)qr-(9)dr 
JO 

+  f  Br(u*  +  9v)qr-(9)dQr. 

Jo 

The  stochastic  integrals  are  defined  pathwise,  so  differentiating  under  the  integrals  gives 
the  result.  Comparing  (3)  and  the  equation  for  we  have  the  following  result  by  variation 
of  constants. 

Lemma  3.2.  Write 

m,t  =  JQ  $*(r-,0 ){~(r,u*)yrq*-dr 

+  jf  ){^{r,u*fjvrq*-dQr 

-  fr(r-MI  +  Br(u*)rlBr(u*)(^ 

Then  zt  =  $*(t,  0)r}G,t- 

Proof.  Using  the  differentiation  rule 

/  &--dq+  /  d$*T]-  +  [$, J]}t. 

’  Jo  Jo 

Because  —  I,  therefore 

=  Jq  (j^(r,u*))vrqr-dr 


(4) 
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+  /o  ^t^ut))v^-dQT 

-  j\l+  Br(«*))-1Br(«*)(^(»-.«*))®r«!L<Wr 

+  /  Ar(u)$*(r-,  0)t/o  r_dr+  f  Br(u)$*(r-,0)r]Q  r_dQr 

Jo  Jo 

+  J  Br(u){—^{r,u*)sjvrq*_dNr 

-  /  B,(«)y  +  'Br(tt*))-1Sr(«*)(^(r,tt*))»rg;_<OVr. 

Now  the  <iiV  integrals  sum  to  0,  showing  that  $*77  satisfies  the  same  equation  (4)  as  2. 
Consequently,  by  uniqueness,  the  result  follows. 

Corollary  3.3.  %(v-*e)\e=Q  =  $*(T,0>7o,:r)]- 

Proof.  «/(w|)  =  i?i [{£,<&(#))].  The  result  follows  from  lemmas  3.1  and  3.2. 

Notation  3.4.  Write  4>*(T,0y  for  the  transpose  of  $*(T,0)  and  consider  the  square 
integrable,  vector  martingale 

Mf.=  |  Yt). 

Then  Mf  has  a  representation  as  a  stochastic  integral 

Mt  =  El[$*(T,0)’e}+  f\rdQr 

Jo 

where  7  is  a  predictable  valued  process  such  that 

rT 


f 

Jo 


Under  a  Markov  hypothesis  7  will  be  explicitly  determined  below 
Definition  3.5.  The  adjoint  process  is 

pt  := 


r  <  00. 
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Theorem  3.6. 

<w«)  I  _  'T 

o=o  Jo 


dd 


=0=JqE1  [(pr-^^ir^-il+Briu*))-^ 

+  (Tr,**(r-,0)(/  +  Br(t<*))-1(||(r,u*))Ur?;_)]</r.  (5) 


Proof.  First  note  that 


( Mt,t]0,T )  =  jQ  (Mr-,$*(r-,0)(^(r,u*)')vrq*_.}dr 

T 

-  f  (Mr-,<f(r-0)(I  +  Br(u,)r1 

X  Br^^ryfjvrq^dNr 
+  J  (lr,W,r-  }iQr 

rp 

+ L  {7r®*(r~,oK^(r,u*'>)vrq*-)dNr 


T 

J  (lr, + 


X  Br^^ry^Vrq^dNr 


Taking  expectations  under  P,  we  have 

dJ(u*e ) 


de 


0=o  =  £71[(A^(T,O)7?o>r}] 

=  £?l[(r(T,0)^»70,T)]  =  MMt^t)}. 


(6) 


Combining  the  last  two  terms  in  (6)  and  using  the  fact  that  Nt  —  t  is  a  P\  martingale,  this 
is 

=  /0  £l[{pr-,(^(r,U*))vr?;_) 

-  (pr-,(I  +  Br{u*))~lBr(u*)(^{r,u*^vrqr-) 

+  (7r,  **(r-  0 )(/  +  Br(u*))~l  (| ®  (r,  )] dr. 
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□ 

Now  consider  perturbations  of  u*  of  the  form  =  u*(t)+6(v(t)—u(t))  for  9  €  [0, 1] 
and  any  v  €  U_.  Then  as  noted  above 


dJ{ue) 


d9 


=  (j\u*),v(t)-u*(t))>  0. 


e=o 


Expression  (5)  is,  therefore,  true  when  v  is  replaced  by  v  —  u*  for  any  v  €  jZ>  and  we  can 
deduce  the  following  minimum  principle. 

THEOREM  3.7.  Suppose  u*  £U_  is  an  optimal  control.  Then  a.s.  in  w  and  a.e.  in  t 

(pj — t  {  -  V  +  }(»r  -  4)5?-) 

+  {-/r,  **(>-,  0 )(I  +  SrtP*))-1  (^(r,  u*j)  (vr  -  u*M-  }  >  0.  (7) 


4.  The  Adjoint  Process 

The  process  p  is  the  adjoint  process.  However,  (7)  also  contains  the  integrand  7.  In 
this  section  we  shall  obtain  a  more  explicit  expression  for  7  in  the  case  when  u*  is  Markov, 
and  also  derive  forward  and  backward  equations  satisfied  by  p. 

Assumption  4.1.  The  optimal  control  u*  is  a  Markov,  feedback  control.  That  is,  u*  : 
[0,T]  x  Rn+1  U  so  that  u*(syql_)  €  U. 

LEMMA  4.2.  Write  8  for  the  predictable  “integrand”  such  that  Apt  =  pt  —pt-  =  <5;  AiV^, 
i.e.,  pt  =  pt-  +  8tANt.  Furthermore,  write  qt _  =  q,  Bt-(u*(t- ,  q ))  =  B*(qt _)  -  B*(q), 
and  Bt(u *  ( t ,  ft))  =  B*(qt).  Then 

6t(q)  =  (1  +  B*((J  +  B*(?))g))-1w_((/  +  B*(„))9)  -  (8) 


Proof.  Let  us  examine  what  happens  if  there  is  a  jump  at  time  t ;  that  is,  suppose 
ANt  =  1.  Then  ft  =  (/  +  B*(q))q.  By  the  Markov  property  and  from  Definition  3.5, 

pt  =  E[D*{t,  Tk){I  +  B'Tk(u*)) . . .  D*(T,  TN)l  \  Yt\ 

=  Pt(<lt)  =  Pt((I  +  B*(q))q) 

=  (J  +  B'(«)')-1P(-((7+B*(9))4), 

and  the  result  follows.  Heuristically,  the  integrand  8  assumes  there  is  a  jump  at  t;  the 
question  of  whether  there  is  a  jump  is  determined  by  the  factor  ANt- 
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THEOREM  4.3.  Under  Assumption  4.1  and  with  8>t  given  by  (8) 


lr  =  $*(r-,0mi  +  B'r(u*))8r  +  B'r(u*)pr_).  (9) 

Proof.  §*(t,0)'pt  =Mt=  Ei[$*(T,0)'i  |  Yt]  =  E\ 0)'£]  +  j\rdQr.  However, 
if  u*  is  Markov  the  process  q*  is  Markov,  and,  writing  q  =  qf,  $  =  $*(i,0), 


I  Y(]  =  EdWiT'tyt  I  q,$] 

=  §fEi[$*(T,t)U\q). 

Consequently,  pt  =  E\[^ {T ,t)' i  |  q]  is  a  function  only  of  q,  so  by  the  differentiation  rule: 

Pt  =  P0  +  J  ~^~(Aqr-dr  +  Bqr-dQr )  +  j  dr 

+  E  (pr-Pr--^Bqr-ANr) 

=  PQ+  f  \-^—-{Aqr—  —  Bqr—)  +  Sr\dr  +  f  5rdQr. 

JO  1  dq  Jo 

Evaluating  the  produce: 


Mt  =  **(tt0Ypt 

=  P0  +  J  $*(r-,0)'  ^^-{Aqr-  -  Bqr_)  +  6r  dr 

+  /V(r-,0  )'%^dr  +  /V(r-,0)'MQr 

JO  dr  Jo 

ft  ft 

+  /  $*(r-t0)'A,pr-dr+  /  $*(r-,0)'Bfpr..dQr 
Jo  Jo 

+  [l  §*(r—,0)'  Br8rdQr  +  f  $*(r-,0)' B'8rdr. 

Jo  Jo 

However,  Mt  is  a  martingale,  so  the  stun  of  the  dr  integrals  in  (10)  must  be  0,  and 

7r  =  9*(r-,0)%  +  S'(4)Sr  +  B’r(u*)V  r_). 
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THEOREM  4.4.  Suppose  the  optimal  control  u*  is  Markov.  Then  a.s.  in  u>  and  a.e.  in  t, 
u*  satisfies  the  minimum  principle 

(pr-,  -  <)?;_)  +  («,,  ^(r,U*)(»r  -  «*)<£-)  >  0.  (11) 

Proof.  Substituting  7  from  (9)  into  (7),  and  noting  jB(J  + 1?)-*  —  (I  +  =  0, 

the  result  follows.  (Substituting  for  B  and  S  gives  an  alternative  form.) 

We  now  derive  a  forward  equation  satisfied  by  the  adjoint  process  p : 

Theorem  4.5.  With  6  given  by  (8) 


Pt  =  ^[#*(2*.  0)'£]  -  f  A^Opr-dr 

Jo 

-  f\l  +  B'r(u*))5rdr  +  fsrdNr. 

Jo  Jo 


Proof,  pt  =  # *{t,0)'Mt  and  this  is 


=  .EM$*(T,0)'fl-  f  A!**' Mdr 

Jo 

-  [tB,V*'MdQr+  f\l  +  B,)~1B,2W*,MdNr 
Jo  Jo 


+  f  **7 rdQr  -  f  B' 
Jo  Jo 


**'7 rdNr 


ri 

+  /  (I  +  B'r^-B'^'jrdNr 

Jo 

=  Ei[$*(T,0)'e\-  f  A’pr-dr  -  f  B'pr-dQr 
Jo  Jo 

+  [t{I  +  B')-1B,2Pr-dNr+  /*((!  +  B')8r  +  B'pr^dQj 
JO  Jo 

-  f  (I +  B’rlB,((I  +  B’)$r  A  B’pr-)dNr 

Jo 


=  Ei{^(j\0)U]  -  /  A! pr— dr 

Jo 

rt  __  ft 

+  /  (I  +  Bf)6rdQr  -  /  B'SrdNr. 

Jo  Jo 


itll*  t  I'kW-iF  U  !■  fl‘,J  '-t11.  ^  1,1 


Therefore,  the  result  follows.  □ 

However,  an  alternative  backward  equation  for  the  adjoint  process  p  is  obtained  from 
the  observation  that  .the  sum  of  the  bounded  variation  terms  in  (10)  must  be  identically 
zero.  Therefore,  we  have  the  following  result  which  appears  to  be  new: 

THEOREM  4.6.  With  8  given  by  (8)  the  Markov  adjoint  process  pt(q)  is  given  by  the 
backward  equation 

f*  +  ■  (A*(*)'  -  B\q)')q  +  A%q)'Vi  +  (I  +  B\q)')St  =  0 

with  the  terminal  condition 

PT=8. 
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Abstract.  -  Time  reversal  is  considered  for  a  standard  Poisson  process, 
a  point  process  with  Markov  intensity  and  a  point  process  with  a  predic¬ 
table  intensity.  In  the  latter  case  an  analog  of  the  Frcchet  derivative  for 
functionals  of  a  Poisson  process  is  introduced  and  used  in  techniques  of 
integration-by-parts  to  obtain  formulate  similar  to  those  of  Follmer  in  the 
Wiener  space  situation. 

Key  words :  Point  processes.  Poisson  process,  predictable  intensity.  non-Markov,  integra- 
tion-by-parts.  Frcchet  derivative. 

Resume.  -  Le  rctourncment  du  temps  cst  considcre  pour  un  processus 
de  Poisson,  un  processus  ponctucl  avec  intensite  markovienne  et  un  pro¬ 
cessus  ponctuel  avec  intensite  previsible.  Pour  le  dernier  cas.  nous  introdu- 
isons  une  sortc  de  derivec  Frcchet  pour  les  fonctionnels  d’un  processus  dc 
Poisson  et  I’utilisons  dans  les  met  bodes  d'integration  par  parties  pour 
obtenir  des  formuies  qui  sont  similaircs  a  tcIIcs  dc  Follmer  pour  la 
situation  brownienne. 
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1.  INTRODUCTION 


The  time  reversal  of  stochastic  processes  has  been  investigated  for  some 
years.  One  motivation  comes  from  quantum  theory,  and  this  is  discussed 
in  the  book  of  Nelson  [II].  The  time  reversal  of  Markov  diffusions  is 
treated  in,  for  example,  the  papers  of  Elliott  and  Anderson  [4],  and 
Haussman  and  Pardoux  [8].  However,  the  first  discussion  of  time  reversal 
for  a  non-Markov  process  on  Wiener  space  appears  in  the  paper  by 
Follmer  [7].  in  which  he  uses  an  integration-by-parts  formula  related  to 
the  Malliavin  calculus. 


In  the  present  paper  an  analog  of  the  Frechet  derivative  is  introduced 
for  functionals  of  a  Poisson  process.  The  intcgration-by-parts  formula  on 
Poisson  space,  see  [6],  is  formulated  in  terms  of  this  derivative  and 
counterparts  of  Follmer’s  formulae  are  obtained. 

In  Section  2  the  time  reversed  form  of  the  standard  Poisson  process  is 
derived.  Section  3  considers  a  point  (counting)  process  N  with  Markov 


intensity  //( N,),  so  that  Q,  =  N,- 


h  (Ns)  els  is  a  martingale,  and  obtains 


Jo 

the  reverse  time  decomposition  of  Q  for  te(0. 1],  Finally,  in  Section  4, 
the  situation  when  h  is  predictable  is  considered  using  the  “Frechet” 
derivative  and  integration-by-parts  techniques  mentioned  above. 


2.  TIME  REVERSAL  UNDER  THE  ORIGINAL  MEASURE 


Consider  a  standard  Poisson  process  N  =  { N,:0^/:g  I }  on  (Q..F,  P). 
We  take  No=0.  Let  {P,}  be  the  right-continuous,  complete  filtration 
generated  by  N.  Let  G,°  =  a{NJ:/g.rg  1 }  and  {G,}  be  the  left-continuous 
completion  of  {G? }. 

The  following  result  is  well  known;  see,  for  example,  Theorem  2.6  in 
(yj.  For  completeness  we  sketch  the  proof. 
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Theorem  2.1.—  Under  P,  N  is  a  reverse  time  G,-quasimartinga!c,  anil 
it  has  the  decomposition: 


Jt 


s 


where  M  is  a  reverse  time  G, -mart ingale. 

Proof.  -  Since  N  is  Markov.  \vc  have,  for  s>0, 


E  [N,  -t  ~  Nr  |  G,] = E  [N,  _t  -  N,  |  N(] 


—  !n. 


(see  [5]  and  [10]).  Thus 


(2.1) 


E  ]  E  [NJ_C  -  N,  |  GJ  |  ds  =  0  (c). 

Jo 

By  Strieker's  theorem  [12],  N,  is  a  reverse  time  G.-quasimartingalc,  Con¬ 
sidering  approximate  Laplacians  we  see  it  has  the  decomposition 

Nl»N1  +  Mf+|'a,<fc  (2.2) 

where  from  (2.1)  and  (2.2),  for  almost  all  t 


X-  lim  -  E  [as  j  G,]  ds 

c  I  0  S 

-lim  'e[N,_c-N,|G(] 

cl  0  t 

=  _  -V,  G 

l 


3.  TIME  REVERSAL  AFTER  A  CHANGE  OF  MEASURE: 

THE  MARKOV  CASE 

Consider  a  process  /;,  =  /;( Nt)  which  satisfies:  There  exist  positive  cons¬ 
tants  A,  K>0  such  that  0<A</i(N,)gK  for  all  /.  a.s. 

Define  the  family  [  A„0gf  g  1  }  of  exponentials: 

*  \ 

A,=  fl  (l+(A(N„-)-«)ANu)cxp(  (1-/»(NuJ)(/h). 

OSu$t  \Jo  / 
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Then  A  is  an  (^-martingale  under  P,  and  is  the  unique  solution  of  the 
equation 

ff 


A,  =  1  + 


A„_  (/i(N„_)-  \)(dNu-dit). 


Define  a  new  probability  measure  P'1  by 

dPh 


dP 


=  A.- 


Then  under  P\  the  process  H 


,=N-[' 

JA 


/i(N„_)r/n  is  an  (^,)-martingale 


(see  [3]).  Let  p(/)  = 


/i(Nu_)r/»  so  that  p  is  positive  and  increasing  in  / 


because  li  is  positive.  Write 


N;=bw 


Vdr 


Lemma  3.1.-  (N,')  is  a  Poisson  process  under  (Cl,.?  ,(&'),  Ph). 

Proof.  -  Since  H,  =  N,-p(/)  is  an  (.^-martingale  under  P\ 
H,'  =  H<,(l)=N<((I)-r  is  an  (.^-martingale  under  P\  By  Ito’s  rule. 


H'2  =  2 


=  ') 


H;.rfH;+Z(AN+w)a 


h;_</h;+n,w. 


Hence  is  also  an  (.^-martingale  under  P\  Therefore.  { N' }  is 

Poisson  by  Levy’s  characterization  (Theorem  12.31  in  [2]).  □ 

Lemma  3.2.  -  N  is  Markov  under  P\ 

Proof.  -  Consider  any  <peCg  (R).  For  /2:.v,  by  Bayes’  formula, 

E[A,|.^.J 
=  E[A'(p(N()|.^,] 

=  E  [A'  (p  (Nr)  |  N  J, 

because  N  is  Markov  under  P,  where 

Aj--  n  d  +(/»(N.)“  I )  A  N„)  exp  f  f(l  -/;(N„))r/A 


Annates  de  Vlnuiiut  Henri  Poincare  •  Probabililcs  ct  Statisliqucs 


TIME  REVERSAL 


361 


Hence 

E>(N,)|.^]=E>(N,)|NJ 

and  N  is  Markov  under  P\  □ 

Note  that 


H,  =  H,  +N,-N,  + 


h(Ns)ds. 


(3.1) 


Thus  H,  is  a  reverse  time  G, -quasimartingale  under  P'1  if  and  only  if  N, 
is.  To  determine  the  reverse  time  decomposition  we  again  investigate  the 
approximate  Laplacians,  as  in  [4], 


Theorem  3.3. 


lim 

c  i  o  s 


-E',[N,_c-N,|G(]=-E'f/i(Nl--l) 


N, 


L 


h(Nu)dti 


N, 


■  (3-2) 


Proof.  -  By  Lemma  3 . 2, 

E*tN,-NI.t|GJ  =  E*[Nl-Nl_,|Nf]. 

Consider  a  bounded,  differentiable  function  <p  on  R  and  its  restriction  to 
Z  (the  range  of  N).  Now 


<p(Nl)  =  <p(N,_t)  + 


(q>(N,_  +  l)-q>(NI_))dN,. 


So 


<P(N,)(N,-Ni_c)  = 


(N,_  -  N, _,)  (q>  (N,_  +  I ) -  q>  (N, .))  <*N, 

C 

<p(Nf_)dN,+  I  A<p(Ns)ANs 

t  r—c<5$/ 

(Ns_-N,_t)(q>(N,_  +  l)-<p(Ns_))^/N5 


+ 


<p(Ns.  +  l)r/Ns 


Since 


H(  =  N,-  r 

Jo 


h  (Ns)  d.s 


r  /vt  \ 


0 
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And 

E"  [q>  (Np  (0  +  1 )  A  (Np  <t))  |  f3  (/)] 


=  X  <p(/r+ !)/»(*) 


P(/r* 


k! 


“  Rf/Vp-noi  / 

=  y  <p  (/)/,(/- ijEMLf - L 

^  /!  p(0 


1  =  0 


.Nr, 


=  El‘l<p(N'm)h(N'(,)-l)-^  P(/) 


=  EA 


=  Eh  <p(N,)/i(N,-l) 


P(0 


PW 


Hence, 


Eh  [<p  (N,  +  1 )  I,  (N,)]  =  E'1  <p  (N,)  h  (N,  -  1 ) 


N, 


mu)du 


.  (3.5) 


Thus  from  (3.4)  and  (3.5), 


iim  E" 

c  i  o 


<P(H) 


(N-H.J 


=  Efc  |"<p  (N,)  h  (N,  -  I )  — — — - 

/i(N  U)du 


or 


Iim  E* 
c  !  0  L  e 


N  -N 


G, 


=  -Eh  h  (N,  -  1 )  N' 


h  (N„)  (hi 


*,  ■ 


N,  .  □ 


By  Theorem  3 . 3  and  an  argument  similar  to  that  in  [4],  we  see  that  N. 
and  hence  H,  is  a  reverse  time  G, -quasimartingale  under  P\  and  it  has 
the  decomposition 


l.  +  l  Mr 


H,=  H,  +  M 

Moreover,  we  have  the  following  expression  for  a,: 

Theorem  3.4.  -  The  integrand  a,  that  appears  in  (3.6)  is  given  by 

a,  =  h  (N.)  -  E"  [/;  (N.  -  I )  ^ - 1 N,]. 

I  li(Ha)du'  J 
Jo 


(3.6) 


-  - r  — 
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Proof.  -  From  (3.1)  and  (3.6), 
E',[H(.c-H(|G,]  =  E',r f  asds\G, 


=  E»[Nl_t-N,|Gl]  +  E* 


h  (Ns)  ds  |  G, 


Thus  for  almost  all  t 


a,  =  lim  -  Eft 
1 1  o  s 


n 


s(!s\G,  =  lim  - E'1  [N,  _t  -  N,  | G,)  +  /; (N,). 
J  c  !  0  £ 


From  Theorem  3 . 3,  a,  has  the  stated  form.  □ 


4.  TIME  REVERSAL  AFTER  A  CHANGE  OF  MEASURE: 
THE  NON-MARKOV  CASE 


This  section  involves  an  integration  by  parts  for  Poisson  processes  which 
is  effected  by  using  a  Girsanov  transformation  to  change  the  intensity  and 
then  compensating  by  a  time  change.  In  contrast,  the  integration  by  parts 
considered  in  [I]  is  obtained  by  introducing  a  perturbation  of  the  size  of 
the  jumps.  The  topic  is  further  investigated  in  [6]. 

Suppose  is  a  Poisson  process  with  jump  times 

T,  a  1,...J„a  I,...  Let  {m, }  be  a  real  predictable  process  satisfying 
{ u, }  is  positive  and  bounded  a.  s. 

For  £>0,  consider  the  family  of  exponentials: 


[I  (l+£«sANs)exp 

Ogsgr 


c  uscis  1. 


Then  (A'}  is  an  {.^l-martingale  with  E[Aj]=  1  (see  [6|).  Define  a  proba¬ 
bility  measure  Pc  on  PP  x  by 


Set 


4>e(0  = 


(ire  u  j  els 
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and  write 


'K(') 


=<pr‘(o=  f 

Jo 


1 


1  +  e  tr 


■ds 


♦.w 


^  t  ^4»c(0- 


Then  the  process  Nj  =  (1)  is  Poisson  on  (Q,  .5%  (^J),  Pc)  with  jump  times 

<pc(T,)  a  1,  .  .  .,(pc(T„)  a  1, . .  .  (see  [6]). 

'< 

For  { u, }  as  above,  set  U,  =  us  ds.  Suppose  gs  (ir)  is  an  {  F,  }-predictablc 

Jo 

function  on  [0, 1].  Then  for  O^x^Tj  a  1, 

&(»’)=£  (4 

and  in  general,  for  T„_,  a  1  <.?^T„  a  1, 

&(“')=£($, T,  a  1,  .  .  .,T„_,  a  1). 

Note  that  by  setting  gs(0,0,  .  . -)=g(s)  for  0g.?^T,  a  1, 

«s((.v-Ti)vO . (r-T„.,)vO),0,0,...)  for  T„.,  a  kigT„  a  1, 

etc.,  such  a  g  can  be  written  in  the  form 

&(»,)“&((*-T,)vO.(*-T2)  vO,...),  *e[0.1].  (4.1) 

Therefore,  we  shall  consider  a  predictable  function  g  of  this  form,  and 
further  assume  that  if 

S~Ss(l  i>'2 - )> 

(]g 

then  all  the  partial  derivatives — 5  exist  for  all  s.  and  there  is  a  constant 

St; 

K>0  such  that 


Oh 


<K  for  all  /,  and  for  all. v. 


(4.2) 


We  now  define  the  analog  of  the  Frechet  derivative  for  functionals  of  the 
Poisson  process. 

Write 


Then 


dz 


«5“&((J“<P«(Ti))  v  0,  .  . . , (s — (pc (T„))  v  0,  . . .). 


£  5 


=  "  I  f &((J-T,)  v  0, . .  .,(.v-T„)  v  0, . .  .) 

|e— O  i=\Vt, 


*  uTdv  1-r.  <5-  (4.3) 
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Define 


\i(d')=  ~  I  ^IT ,<sSTi(^) 

1=1 


where  6T|  is  the  point  mass  at  T{.  Then 

=  j  urdr\x(dt) 
c  =  0  Jo  Jo 


where 


Write 


Note  that 


ds 


I0Sr 


\i  ([i\s])u,dr 

rs  00  a 

“  Z  IrgTi<s“Wr^' 

Jo  i-  1 

=  D&(.,[r,.y])i/rrfr, 

Jo 


dsx^])=-iuTi.3. 

i-l  ot{ 


D&(.,U)- 


Dgs(..[r,s])urdr. 


i~  i  <7/y 


it,  r/r. 


(4.4) 


Definition  4.1.  -  A  process  {&}  of  the  form  (4.1)  is  said  to  be 
differentiable  if  it  satisfies  (4.2)  and  (4.3)  for  all  u  satisfying  (i)  and  (ii) 
above,  and  for  all  j.  We  call  Dgs(. ,  U)  the  derivative  of  gs  in  the  direction 
U.  It  is  of  interest  to  note  that  this  concept  of  differentiability  of  a  function 
of  a  Poisson  process  is  an  analog  of  the  Frechet  derivative  of  a  function 
of  a  continuous  process.  See  Follmer  [7],  where  similar  formulae  arise 
using  (iie  Frechet  derivative. 


Ann, ties  Je  I'lnslitul  Ham  Poincare  -  Probability  ct  Statistiqucs 


TIME  REVERSAL 


367 


Now  suppose  { hs }  is  a  bounded,  { F,  }-predictable  process  of  the  form 
given  by  (4. 1),  which  satisfies: 

(a)  h  is  differentiable  in  the  sense  of  Definition  4.1. 

(b)  —  exists,  and  there  exists  a  constant  A>0  such  that  —  <  A  for 

ds  ds 

all  x,  a.s. 

(c)  There  are  constants  B>0,  C>0  such  that  0<B</is<C  for  all  s, 
a.s. 

It  is  easy  to  check  that  /js=/is((.v-T,)  v  0.  (a'-T2)  v  0,  . .  .)  is  predict¬ 
able.  Consider  the  family  of  exponentials: 


G,=  n  (I+(/ts-l)ANs)exp  (1-/0* 


=(  n  (i-^o* 

\0ST;gi  /  \Jo 


Then  {G,}  is  a  martingale  with  E[G,]=1.  Since  for  each  fixed  to,  if 
T„_ ,  (to) <  t  gT„ (to),  G,  is  a  function  of  (/,  T,  (to),  .... T„_ ,  (to)),  we  see  as 
above  that  G,  can  be  considered  to  be  of  the  form 

G=G,((/-T,)  vO,...,(/-T„)  vO....). 

Theorem  4.2.  -  (G,)  defined  in  (4.5)  is  differentiable  in  the  sense  of 
Definition  4.1. 

Moreover, 

DGj (.,U)Gfl  =  |  Y,«,Gr‘* 


pi  l  f  00  a/  i  | 

I  I(T;<s)~  +  D/t5(.,[r,.y])  —  </N5 ur dr 

JoJrL*  ;=  i  Stj  J/is 

-J  |  Dhfi.,[r,s])dsurdr,  a.s.  (4.6) 


Y,=  “  Z  IsgTfgi  T"Gi  ((1  “T,)  v  0, . .  .,(1  — T„)  v  0,  . . .). 

i=I  otj 

Proof.  -  The  first  identity  follows  from  the  definition  and  properties 
of  the  derivative.  To  determine  DG,(.,U)  we  calculate  the  derivative  of 
G'  at  l-O.  Write 

^=M(*-< Pt(T,))  v  0,  .  .  ,,(s-(pc(T„))  v  0 - ), 
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SO 


Gf=  FI  (l+(/i5-l)AN,i(s))exp(  |  (I  -IQds 

n 

(1  -hcs)  els 


=(  n 

;4(t,))cxP 

OSpt(T,-)gi 

-<  n 

/'7(T,))eXP 

OST.S^W 

(l  . 


Then 


logG'-  £  lT1g<.t(t)l°8^c(T1)  + 


(!  -IQ ds. 


(4.7) 


Differentiate  (4.7)  with  respect  to  s,  and  then  set  e  =  0,  to  see 


DG,(.,U)l=I^ITiS( 

G,  i=  1 


d/iT.  ft, 


‘Tj 

17 


it,  dr 


'~ldhT.(n, 


+77u. “ 


ii,  dr 


D  /;5  ( . ,  U)  ds,  a.s. 


From  (4.4)  this  is 


-IN 


dhr.  ft, 

17 


u,  dr 


i-l 


_  dhj. 

+  I  — 

A  d‘j  j 


fT.- 


i/r  clr  +  D  /iT  ( . ,  U) 


Dhs(.,U)ds= 


4" 


7 


't 

Jo 

_  ds  . 

1 

f,T| 

ii,  dr 


dr  +  D/i,  ( . ,  U) 


r/N. 


_J  s 

n 


D/;s(,U)r&.  (4.8) 


(Formally,  the  differentiation  of  the  indicator  functions  ITi;.Vt(I)  introduces 
Dirac  measures  5(/-T,-.)  However,  P(T,=  /)=0  and  we  later  will  take 
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expectations,  so  these  can  be  ignored.)  From  (4.8), 
DG1(.,U)Gr‘=  f‘{^  \\dr+  £  I,TJ<„^ 

Jo  (“Jo  j  =  1  Otj 

+  D/is  ( . ,  [r,  .v])  ur  dr  1  y  r/N,  - 

Jo  J  I's 


urdr 


Dhs  ( . ,  [r,  .9])  ur  dr  ds 


n 


r  1 


*OSrSs£ 1 


fdA,  1  "  .  a/ts  1 

)  "T"  Ur  .  4"  £  Vr,  <s)  T  Ur  , 

{  ds  hs  Jsl  J  •dtj  hs 


+  D hs  (.,[r,  s])  ur  ~  )  dr  dNs  - 

l‘s)  Jo  J 


M  •>['',  *])  urdrds 


M 

f1 

Jo  * 

r 

^+IItT><s»f!i+D/,J(.,[,-,.9]) 

ds  ;=  1  Otj 


n 


~  dHs  ur  dr 

K 

D  hs(.  ,[r,s])ds  urdr, 


which  is  (4 . 6).  □ 

Consider  the  family  of  exponentials  defined  by  (4 . 5)  and  define  a  new 
probability  measure  P'“  on  -9  x  by: 

dPh 


dP 


Then  ( see  [3])  the  process 


=  G,. 


-1 =  N,  -  f'  hs  ds 
Jo 

=Q  -  ['(/»,- 

Jo 


(4.9) 


1  )ds. 


where  Q,  =  N(-/,  is  an  (.^-martingale  under  Ph.  We  want  to  show  that 
Z,  is  a  reverse  time  G, -quasimartingale  under  P\  having  the  decomposition 


Z,  =  Z[  +  M,+ 


From  (4 . 9),  we  can  write 


Z,  =  Z1  +  Q,-Q,+ 


a.  ds. 


(Its-  1 )  r/.v. 


(4.10) 


Now  for  almost  all  1 


lim  -  E*  I  (hs  -  1 )  ds  |  G,  =  E*  [/i,  -  1 1 GJ. 
c  I  0  E  _  J  r  —  c 
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Hence,  to  show  that  Z,  has  the  decomposition  given  by  (4.10),  it  again 
suffices  to  consider  approximate  Laplacien  as  in  [4]  and  show  that 

iim  -E'1[Q,-C-Q,|G(] 
e  1  o  £ 


Theorem  4.3.  -  For  almost  all  / e [0, 1) 

lim  - E* [Q, - Q, _c | G(]  =  - E'1  [Q,  +  a, | GJ - EA [bt j G,]  (4.11) 

e  l  0  £  t 


f'fT 8hr  *  ,  dhr 
JoJj  l  dr  j- 1  G,j 

-  j  rt  ri 

+  D//r(.,M)  —tlNrds-  D/;r  ( . ,  (x,  r])  drcls 

J/'r  JO  Js 


n  r  ph  °°  (111  l!  f 1 

b,  =  +  I  ,rr,<M^r  +  D/ir (..[/. ,])  - ,/Nr -  D hr (.,[/. r}) dr. 

l  dr  J=1  dtj  J  K  J. 

Proof.  -  First  we  note  that  if  H  ((1  -T,)  v  0,  .  .  .,(1  ~T„)  v  0,  .  .  .)  is 
a  square  integrablc  functional  and  its  first  partial  derivatives  arc  all 
bounded  by  a  constant,  then,  using  a  similar  argument  as  in  [6],  we  have 
the  integration  by  parts  formula 

E  =»E[DH(.,U)]  (4.12) 

where  DH(.,U)  is  the  derivative  in  direction  U  of  Definition  4.1. 

A  direct  consequence  is  the  product  rule 

E  FH^  '  us(Iq)j  =-E[FDH(..U)]-E[HDF(.,U)].  (4.13) 

Let  H  =  G,  be  the  Girsanov  density,  then  (4.13)  becomes 


{fPmq5  =- 

.Jo 


E',(DF(.,U))-E',[FGr1DG1(.,U)].  (4.14) 


Now  fix  /oe(0, 1).  Write  Tn(/0)  for  the  fc-th  jump  time  of  N,  greater  than 
t0.  Suppose  F  is  a  bounded  and  G,0  measurable  function.  Furthermore, 
we  suppose  that  F  is  a  differentiable  function  (in  the  sense  of  Definition 
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4. 1)  of  the  form 

F((1-T,(/o))vO,...,(1-T*(/0))vO,...X 

and  that  the  derivatives  of  F  are  bounded.  Then  the  measure  DF(.,J/)  is 
concentrated  on  [f0,  1]  and  (4.14)  holds  for  such  an  F.  Take 
.,,0)(.?)  in  (4. 14).  For  such  an  F 

DF(.,U)=  I  DF(.,[r,  \])dr 

JtQ-t 

=  '°  DF(.,[/0,  l])r/r 

=  eDF(.,[/0, 1]). 

Therefore,  we  have  from  (4. 14) 

E*[(Q,0  — Q,0_t)  F]=  -sE,I[DF(.,[t0. 1])] 

r  f'o  x  8G 

+  E'1  FG,-1  IWi  ~ds  .  (4.15) 
L  0lt 

From  (4. 15),  for  almost  all  / 


Hm  -[(Q,0-Q,0-C)F]=  -H"[DF(.,[t0, 1])] 

c  l  0  6 

4-E'jFGr1  (4.16) 

1-1  vti  _ 

Using  (4.15)  again  with  e  =  *0  =  /,  we  have 
-E‘[DF(..[M])]=-E*[Q,F] 

t 

1  T  f<  60  prt 

-  -  Eh  FGJ"5  .  (4.17) 

1  L  Jo  i=  I  St, 


Now  let  hs  =  I(0  ,|(.y)  in  Theorem  4.2  to  obtain 


Y  ^sgT,<l 


G,  1  els 


=  ’*  Y  I( Tj<r ) “  +  D/ir ( ■ , [s, /"])  y r/N, ds 

JoJsL^-  j=  1  St,  J:r 


D/ir  ( . ,  [.v,  /’])  dr  ds. 


Vol.  26,  n'  2-1990. 


372 


R.  J.  ELLIOTT  AND  A  H,  TSOI 


Hence  (4.17)  becomes 

- E" [DF  ( . , [/,  !])]=- E* [Q,  F]  +  i E* [a, F]. 

t  t 

Now  take  i/s=l(,_t  ,j  (s)  in  Theorem  4.2  to  obtain 

z 

I  -C  \i=  1  Ot;  ] 


(4.18) 


'(  pi 
t  -c  Js 


,  ^+1  WolT+DAr(-,M 

IBr  jsl  dtj 


-dN.ds 


D/ir  ( . ,  [s,  /■])  dr  ds.  (4.19) 


-E* 


Multiply  both  sides  of  (4.19)  by  F,  and  then  take  expectations 

f  (r  i  S 

LU-,  w‘ 

Divide  both  sides  of  (4.20)  by  s,  and  then  let  0,  *o  obtain  for  almost 
all  / 


'j  Gj" 1  ds 

=  Eh 

1 - 

tc, 

<T 

u- 

i _ 

) 

L  J<-c  J 

.  (4.20) 


-E" 


=E'-[ft,F].  (4.21) 

Combining  (4. 16),  (4.18)  and  (4.21),  we  have 

lim  -  E"  [(Q,  -  Q, .  t)  F]  =  -  E'1  [(«,  +  Q,)  F]  -  EA  [/;,  F] . 

t  J  0  E  I 

Thus  we  have  proved  (4. 1 1).  □ 

As  a  consequence  of  Theorem  4 . 3.  Z.  is  a  reverse  time  G,-quasimartin- 
gale  having  the  decomposition  given  by  (4.10).  It  follows  immediately 
that  the  integrand  a,  in  (4. 10)  is  given  by 


at = Efc  \b, + /i,  -  1 1 GJ  -  -  Eh  [a,  +  Q,  |  GJ. 
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Abstract 

Stochastic  integrals  with  respect  to  a  martingale  X  often  involve 
a  predictable  process  integrated  against  the  continuous  martingale 
component  Xc  together  with  terms  which  are  integrals  of  the  com¬ 
pensated  random  measures  associated  with  the  jumps.  The  latter 
are  related  to  ‘optional’  stochastic  integrals.  The  main  result  of  this 
paper  relates  such  a  stochastic  integral  with  the  sum  of  a  predictable 
stochastic  integral  of  X  and  an  orthogonal  martingale.  The  result 
has  applications  in  the  hedging  of  contingent  claims  in  finance. 
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1  Introduction 

For  a  real  local  martingale 

Xt  =  X0  +  Ml  +  Mf 

write  n  =  fix  for  the  random  measure  associated  with  the  jumps 
of  X  (see  Jacod  [7]),  and  v  —  nF  for  its  predictable  compensator. 
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Consider  a  local  martingale  of  the  form 

Nt  =  N0+  f  <psdM%  +  f  f  ip3{y)(y.{dy,ds)-v{dy,ds))  (1) 

JO  JO  JR 

where  <p  and  ip  are  suitable  integrands.  Projecting  N  on  X  we  can 
write 

Nt  =  N0  +  [\,dXs  +  Tt  (2) 

Jo 

where  T  is  a  local  martingale  orthogonal  to  X ,  in  the  sense  that  the 
product  T.Y  is  a  local  martingale.  In  this  paper  our  purpose  is  to 
determine  explicit  formulae  for  7  in  terms  of  <p  and  ip,  both  in  the 
general  case  and  for  Markov  diffusions  with  jumps. 

These  results  have  applications  in  the  hedging  of  contingent  claims 
in  finance.  See  [5]  and  [6].  For  example,  suppose  that  the  martingale 
Xt  represents  the  price  of  some  asset  at  time  t  and  that  for  T  >  0 
a  contingent  claim  is  given  by  H(Xt),  where  H  is  a  function  such 
that  H(Xt)  is  a  real,  square  integrable  random  variable.  Suppose  at 
time  t  we  invest  amount  &  in  the  asset  and  an  amount  r/t  in  a  riskless 
bond  with  zero  interest  rate  and  price  Y  =  1.  Then  the  value  of  our 
portfolio  at  time  t  is 


Vi  =  StXt  +  VtY  =  ZtXt  +  r)t. 

We  assume  £  is  predictable  with  respect  to  the  filtration  {F(}  gener¬ 
ated  by  X,  and  7  is  adapted.  The  accumulated  gain  from  the  asset 
price  fluctuations  up  to  time  t  is  the  stochastic  integral  J^,dX3. 
Then  the  cost  accumulated  to  time  t  by  using  the  investment  strat¬ 
egy  (ZuVt)  is 


ct  =  vt 


0  <  t  <  T. 


We  want  our  investment  strategy  to  duplicate  the  contingent  claim, 
so  for  a  strategy  (£,  77)  to  be  admissible  we  also  require 


Vt  =  £tXt  +  Vt  =  H(Xt )• 
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Now  suppose  H(Xt )  can  be  represented  as  a  (predictable)  stochas¬ 
tic  integral 

H(Xt)  =  E[H(Xt)}  +  ftfdX,  a.s.  (3) 

Jo 

for  some  (predictable)  integrand  Then  let  us  take  an  investment 
strategy  (£,  rj)  and  value  process  V  defined  by: 

6  =  £?,  rjt  =  Vt-  £tXt 

and 

Vt  =  E[H(XT)]+  fgdXa. 

Jo 

We  have  Vp  =  H (Xp),  so  the  strategy  is  admissible  and  for  all 

<e[o;r] 

Ct  =  CT  =  Co  =  E[H(Xp)]. 

That  is  the  strategy  is  self-financing  because,  apart  from  the  ini¬ 
tial  cost  Co  =  E[H(Xp)],  no  additional  costs  arise  and  no  risks  are 
involved. 

Conversely,  if  there  is  a  self-financing  strategy  (£,77), 

Vt  =  Ct+  UtdXs  =  Co  +  i\,dX„ 

Jo  Jo 

so  Vj  is  a  martingale.  Therefore, 

V*  =  E[Vt  I  Ft]  =  E[H(Xt)  I  Ft] 

and 

V0  =  Co  =  E[H(XT)], 
so  the  martingale  V  has  the  representation 

Vt  =  E[H(X r)]+  fadX,.  (4) 

Jo 

The  existence  of  a  self-financing  strategy  is,  therefore,  equivalent  to 
the  representation  of  the  martingale  E[H(Xt)  \  Ft]  in  the  form  (4) 
for  some  predictable  integrand  f .  In  general,  a  representation  in  this 
form  is  not  available.  In  this  case  we  can  proceed  as  follows  (cf.  [5] 
and  [6]). 
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Definition  1.1  An  admissible  investment  strategy  (f,  rj)  is  said  to 
be  mean  self-financing  if  the  corresponding  cost  process  C  is  a  mar¬ 
tingale.  That  is,  for  t  <T, 

E[CT  -  Ct  \  Ft]  =  0  a.s. 

In  tliis  case,  by  definition 

Vt  =  Ct  +  Jj,dX,  =  E[CT  |  Ft]  +  JjsdXs. 

Therefore,  V  is  a  martingale  for  an  admissible  mean  self-financing 
strategy.  Consequently, 


Vo  =  Co  =  E[H(Xt)] 


and 

Vt  =  E[H(Xt )  |  fi]  =  E[H(Xt))  +  Kt4-  i\adX» 

Jo 

where  Kt  is  the  martingale  E[Ct  \  Ft ]  -  Co.  Note  that,  if  (£,/?)  is 
an  admissible,  mean  self-financing  strategy,  Vt  =  E[H(Xt)  \  Ft]  is 
independent  of  £.  However, 

Ct  =  Cf  =  Vt-  l\tdXt 
Jo 

does  depend  on  £,  as  does  K  above.  Therefore,  Kf  =  E[C f  |  jPt]  -  C$ 
and  each  admissible,  mean  self-financing  strategy  (£,t?)  gives  rise  to 
a  decomposition: 

Vt  =  E[H(Xt)  I  Ft]  =  £[ff(XT)]  +  Kl  +  j\sdX,.  (5) 

Jo 

Definition  1.2  For  each  admissible  mean  self-financing  strategy  the 
remaining  risk  is  defined  to  be 

4  =  -e[(4-c1£)2|f,i. 
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Consider  the  unique  Kunita-Watanabe  decomposition 

Vt  =  E[H(XT)]  +  Tt+  fgdXs 
Jo 

where  T  is  a  martingale  orthogonal  to  X  and  g  is  a  predictable 
integrand.  Now  define  an  investment  strategy  (g,g)  by  putting 

Vt  =  E[H(XT)}  +  Tt+  fgadXa  and  gt=Vt~gtXt.  (6) 
Jo 

Then  (g,g)  is  an  admissible,  mean  self-financing  strategy  which 
minimizes  the  remaining  risk  Rt.  To  see  this  note  that  for  any  other 
admissible,  mean  self-financing  strategy  (£,  i ]): 

4  -  Cf  =  K%-  Kf. 

However,  from  (5)  and  (6) 

T  j1 

4  -  I(f  +  J  gdX,  =  lT  -  Tt  +  J  gdX3. 

Therefore,  because  T  is  orthogonal  to  X , 

T 

Ei(c(T-clf  | «]  =  £!(rr-r,)2 1  (£-( ,?nx,x),  |  f] 

and  this  is  minimized  when  £  =  g.  Consequently,  the  unique  admis¬ 
sible,  risk  minimizing  investment  strategy  is  (£*,»?*),  where  g  is  the 
predictable  integrand  arising  in  the  representation  (6). 

This  discussion  indicates  why  decompositions  such  as  (6),  to¬ 
gether  with  an  explicit  formula  for  the  integrand,  are  of  interest 
in  finance.  Representations  such  as  (1)  arise  when  the  asset  price 
Xt  =  Xo+Mf+Mf  also  involves  random  disturbances  of  jump  type. 
In  that  case,  a  contingent  claim  typically  admits  a  representation  as 
in  (1) 


H(Xt)  =  E{H(Xt)}+  t 
Jo 


rT  r 

Jo  ds)-v{dy,  da)). 

(7) 
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Then  <f>,  and  ips(y)  for  each  y  6  R,  (or,  at  least,  for  each  y  in  the 
support  of  /i  =  nx),  must  represent  amounts  invested  in  different 
assets  in  order  to  duplicate  (i.e.,  represent)  the  claim  H{Xt)-  How¬ 
ever,  if  the  only  assets  available  are  X  and  Y  =  1,  we  must  consider 
the  alternative  representation  (2) 

H(Xt)  =  E[H(Xt)}+  [Ty,dX3  +  rv 

Jo 

Then  7  will  generate  the  risk  minimizing  mean  self-financing  invest¬ 
ment  strategy  described  above. 

In  particular,  even  though  (after  a  Girsanov  change  of  measure) 
the  Markov  diffusion  process  considered  by  Aase  in  [1]  is  complete  in 
a  mathematical  sense,  that  is,  contingent  claims  have  a  representa¬ 
tion  of  the  form  (7),  it  is  not  complete  in  the  financial  sense;  that  is, 
they  do  not  necessarily  admit  a  decomposition  (3).  To  replicate  the 
claims  in  Aase’s  model  an  uncountable  number  of  additional  artificial 
assets  would  be  required.  Clearly  this  is  not  realistic. 

Orthogonal  martingale  representation  after  a  Girsanov  change  of 
measure  will  be  discussed  in  another  paper. 


2  Orthogonal  Projection 

Consider  a  real  local  martingale 


Suppose 


and 


Xt  =  X0  +  M?  +  Mf. 


li  =  nx(dy,dt )  =  £  J{A*<5io}6(afAA:.)(dy,dt) 

i>0 


v  -  v(dy,dt)  -  nP. 

Write  {Ft}  for  the  right  continuous,  complete  filtration  generated 
by  X.  Consider  a  process  N  which  is  a  stochastic  integral  of  the 
form 

Nt  =  No  +  I  <f>,dMZ+  f  f  ij}s(y){fi(dy,ds)  -  v{dy,ds))  (8) 
Jo  Jo  Jr 


Orthogonal  Martingale  Representation 


7 


for  suitable  integrands  <p  and  ip.  What  we  wish  to  do  is  write 

Nt  =  N0+  l\,dXa+Tt  (9) 

Jo 

where  7  is  a  predictable  integrand  and  T  is  a  local  martingale  or¬ 
thogonal  to  X.  Prom  Jacod  [7]  we  know  that  the  stochastic  integral 
/  f  ip(dp-du)  in  (8)  is  related  to  an  optional  stochastic  integral  with 
respect  to  Md  =  fofny(fi(dy,ds)  -  v(dy,ds))-y  consequently  we  are 
relating  the  optional  integrals  in  (8)  to  the  predictable  integral  in  (9). 

Proposition  2.1  Assume  there  is  a  reference  measure  v  =  v(w,  ds ) 
such  that  ( Mc .  Mc)  and  X  are  absolutely  continuous  with  respect  to  v, 
where 

i/(dy,ds)  =  m(sydy)X(ds). 

Write  p3  =  d(Mc,Mc)/dv  and  A,  =  dX/dv.  Then  if  Nt  is  the  mar¬ 
tingale  given  by  (8),  the  process  7  in  (9)  is 

_  P»<ps  +  A,  fR  yj>s(y)m(s,dy) 

P>  +  A,  fR  y2m(s,  dy) 

Proof.  Note  if  7  is  the  predictable  integrand  of  (9) 

At  =  t*isdX.=  [\adMf+  (\3dM? 

Jo  Jo  Jo 

=  I  73dM4c+  /  j  -y3y(n(dy,ds)-  u(dy,ds)). 

Jo  Jo  Jr 

Prom  (8)  and  (9) 

r t  =  /  ((pi  -  7 z)dMcs  +  /  [  (ips(y)  -  i3y)(p(dy,ds)  -  u(dyyds)). 
Jo  Jo  Jr 

The  martingales  T  and  X  are  orthogonal  if  (r,  X]  is  a  martingale 
(see  Dellacherie  and  Meyer  VIII.41,  [3]).  However,  writing  X  as 

Xt  =  Xo  +  Mf  +  [  f  y(n(dy,ds)~  u(dv.ds)) 

JO  JR 


(10) 

(11) 
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we  have 

[r,X]i=  [  ( <f>s-'ys)d(MctMc)a+  j  [  y{i>a{y)-'iiy)n(dy,ds) 

Jo  Jo  Jr 

=  /  (<t>$  ~  la)p>ds  +  /  /  y(rl>t(y)-7ay)(n(dy,ds)-v(dy,ds)) 

Jo  Jo  Jr 

+  I  y(Mv)  ~  7 ay)m(s,dy)Xads. 

Jo  Jr 

This  is  a  martingale  if  and  only  if 

/  (4>a  - /ls)pads  +  [  [  y{-4>a{y)  -  'fay)m(s,dy)\ads 
Jo  Jo  Jr 

is  the  null  process,  which  is  the  case  if  and  only  if  the  integrand  is 
zero.  Therefore, 


(<f>a  -  7 s)pa  +  Xa  y(i>3(y)  -  jay)m(sydy )  =  0 
Jr 

and  (10)  follows. 

Remarks  2.2.  1)  If  Md  =  0,  so  that  n  =  v  =  0,  then  7  =  <f>. 

2)  Note 

/  yMy)m(s,dy)  =  E[AXaANa  \  F„] 

Jr 

=  p(AXAN)s 


j  y2m(s,  dy)  =  E[ AX2  |  fs_]  =  p(AX2)a, 

Jr 

where  p(  )  denotes  the  predictable  projection. 

3)  If  Mc  =  0, 

IRyUy)m(s,dy)  E[AXaANa  j  i^,_]  p(AXAN), 
13  fRy2m(s,dy )  E[AXj\Fa-}  p(AX*)a 


so  7  can  De  interpreted  as  me  regression  01  me  jumps  ox  rv  on  me 
jumps  of  X. 

4)  With  appropriate  interpretation  of  products  as  tensor  products 
in  Rm ,  the  same  expression  for  7  is  valid  when  X  is  an  m-dimensional 
martingale. 
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3  Representation  Results 

Consider  a  real,  local  martingale 

Xt  =  X0  +  Mct  +  Mf 

and  let  p  =  px ,  u  =  pp.  Suppose  F  €  C1'2,  the  space  of  functions 
continuously  differentiable  in  t  and  twice  continuously  differentiable 
in  x.  Then  the  differentiation  rule  gives  (see  Jacod  [7]) 

fldF  NdF 

F(t,Xt)  =  F(0,X o)  +  Jojj(s,X,-)ds  +  yo  ~fa(s>X»-)dMZ 

+  f  [  ( F(s,x, _  +  y)  -  F(s,xs-))(p(dy,ds)  -  u(dy,ds )) 

Jo  Jr 

(12) 

+ J‘Jr(As-x-  +  #)-  F(a.X.-)  -  da). 

Suppose  X  is  Markov.  For  a  time  T  >  0  and  an  integrable  C2  func¬ 
tion  //(■),  consider  the  random  variable  H(Xt)  and  the  martingale 

Nt  =  E[H(Xt)  |  Ft]. 


Because  X  is  Markov 

Nt  =  E[3(Xt)  |  Xt]  =  V(t,Xt),  say. 

The  following  representation  result  appears  to  be  part  of  the  folklore. 

Proposition  3.1  Suppose  V  is  C1'2,  that  is  continuously  differ¬ 
entiable  in  t  and  twice  continuously  differentiable  in  x.  Then  the 
martingale  N  is  given  by  the  stochastic  integral  representation 

Nt  =  E[H{XT)]  +  J^(s,X,-)dM? 

+  /7  (n^Xs-  +  y)-V(s,x,.)) 

Jo  Jr 

x(p(efy,<fs)  —  vldy^ds)). 


(13) 
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Furthermore,  suppose 


(Mc,Mc)t 


and 


v(dy,ds)  =  m(s,dy)Xsds. 


Then  V  is  the  solution  of  the  backward  Kolmogorov  equation 


8V  1  d2V 
ds  2  dx 2 


Ps 


+ JR(V(s,Xt.  +  y)~  V(s,Xt.)  -  s,X'-)y)m(s,dy)\ , 

=  0  (14) 


with  terminal  condition 


V(T,Xt )  =  3(Xt). 

Proof.  The  result  follows  by  expanding  V(t,Xt)  by  the  Ito  rule 
(12)  and  observing  that,  because  V(t,Xt )  =  Nt  is  a  martingale,  the 
sum  of  the  bounded  variation  terms  must  be  the  null  process. 

Remarks  3.2.  Often,  (see  Example  3.4),  the  differentiability  of  V 
follows  from  flow  properties.  In  the  pure  jump  case  only  increments 
of  V  enter  in  (13). 

Corollary  3.3  Write  AV,(y )  =  F(s,  +  y)  -  F(s,  Xs.). 

Then  from  Proposition  2.1  Nt  can  be  written 

Nt  =  E\H{Xt)}  +  l\tdX,  +  Tt 

Jo 

where 

^  _  p>  +  X>ky^v»{y)m^dy)  /15x 

p»  +  KJRy2m(s,dy) 

and  F  is  a  martingale  orthogonal  to  X. 
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Example  3.4.  Suppose  X  is  a  Maikov  diffusion: 

Xt  =  X o  +  f g(s,X,-)dBt 
Jo 

+  f  [  h(s,Xt-,y)(p.(dy,ds)-P(dy,ds)).  (16) 

Jo  Jr 

Here  ft  is  a  random  measure  and  v  =  jip\  note  ji  is  not  fix .  Suppose 
u(dy,ds )  =  m(s,Xa-,dy)\t(Xa-)ds.  Write  £r,t(a:)  for  the  solution  of 
(16)  starting  at  time  r  in  position  x,  so  that 

&,«(*)  =  *  +  j(  g(s,£r,a-(x))dBa  (17) 

+  J  J  h(st{rt'-(x)ty)(fi(dytds)-i>(dy,ds)). 

Suppose  p,  h,  m  and  A  and  their  first  two  derivatives  in  x  are  mea¬ 
surable  with  linear  growth  in  the  x  variable.  Then  from  the  theory 
of  stochastic  flows,  see  [2],  it  is  known  there  is  a  set  A  C  ft  of  mea¬ 
sure  zero  such  that  the  map  (r,f,x)  ->  &.,*(*)  is  twice  differentiable 

in  x  with  derivative  =  Dr,t •  Again  write  {Ft}  for  the  right 

continuous  cr-field  generated  by  X  and  suppose  H  is  a  C2,  integrable 
function.  For  T  >  0  consider  the  right  continuous  martingale 

Nt  =  E[H(So,t(xo))  I  Ft) 

=  E[H{fa{Xt))\Xt]  =  V{t,Xt). 

From  the  differentiability  of  the  flow 

9^M)  =  E{f{Xr)D,,T\Fl}. 


Substituting  in  (13)  and  (15)  we  have  the  representations 

Nt  =  E[E(XT)}  +  j*E[^-(Xr)D0>T  \  Fa.]D^_g(s,Xt.)dBa 

if  f  f  r>r  j ne  v  i  _  ufc  _/  v  \\  I  v  1 

'  /  /  VS«-,Tv^»-  T  yjj  ~  ^  VS a-,l  JJ  I  ■‘'■S-I 

J  0  J  R 

*h(s,  6,«-(*o),  y)(/K<fy, da)  ~  Hdv, ds)) 
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and  from  Corollary  3.3  this  is 

=  E[3(Xt)]  +  [\sdXa  +  rt 
Jo 

where 

7.  =  {g<,s,X.-)‘‘E[^-(XT)Do,T  | 

+A,(JT,.) ljiE[E(t,-,T(X,-  +  y))  -  £({,-, T(X,.))  |  X,.) 

xh(s,  to,a-(x0),  y)m(s,  to,a-(x0),  at/)} 

xJRy2h(s,Zo,*-(xo)iy)2m(s,to,s-(xo),dy)}  \ 

Example  3.5.  The  random  measure  in  (16)  could  be  a  Poisson 
random  measure.  However,  for  simplicity  suppose  it  is  a  finite  sum 
of  independent  Poisson  processes  Nj,  i  =  1, . . . ,  n  with  time  varying 
jump  sizes  a\  and  intensities  A*.  Suppose  g($,X,J)  =  crX and 
h(s,Xt~,y )  =  X*_,  so  that  Xt ,  (representing  an  asset  price  under 
a  ‘risk  neutral1  measure),  is  given  by  the  following  “log  Poisson  plus 
log  normal”  equation 

Xt  =  X0  +  cr J* X,..dB,  +pJ*X,-ai,(dNi  -  A «>).  (18) 

Suppose  for  an  integrable  C 2  function  3 ,  3{Xt)  represents  a 
contingent  claim  depending  on  the  asset  price  at  time  T  >  0.  Then 

Nt  =  E[H(XT)\Ft] 

=  E[E{Xt))  +  [t^dxs  +  rt 

Jo 


7*  - 


MlB[%(xT)D0,T  I  F.-Woi-  +  E  W.v 

Ml  + 1  KM2 

«'= 1 


where 


Orthogonal  Martingale  Representation 
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and 

a;v  =  +  «*))  -  m-rfx,.))  |  x,.]. 

Prom  (18) 

Xt  =  t,-,t(Xs-)  = 

X,.  exp  (<r (B,  -  B,)  -  ^-(t  -  s)  -  53  J  a^X'.di') 

X  n  (1  +  4AJV*).  (19) 

3<r<t 

SO 


£a-,T  =  A),r  ■  DQiI_  = 

exp  (<r(5t  -  B3)  -  ~-(t  -  s)  -  ]T  f  a‘ A‘  ds) 

*  i=i  ■'* 

x  na+<^)- 

$<r<t 

Suppose  H(Xt )  is  a  call  option  of  the  form  H(Xt)  =  (Xt  -  K)+. 
Then  H  is  not  C2  but  is  the  limit  of  the  smooth  functions  Hc(Xt )  = 
\(Xt  -  K  +  \/(Xt  -  X)2  +e);  using  approximation  arguments  it  is 
shown,  for  example  in  [4],  that  the  above  theory  applies  to  H.  Now 

^r( Xt )  =  Ixt>k ,  so 

e[~(Xt)Dqj  I  =  E(Ixt>kD,-,t  |  X,_] 

and  because  the  B  and  JV*  are  independent  this  can  be  evaluated  as 
in  [1],  giving  a  Black-Scholes  type  formula.  Similarly,  with  £s-,t(X3-) 
given  by  (19)  A^V  can  be  calculated. 
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